Issues in Multimedia Synchronization over Broadband

Networks
S. Baqai M. F. Khan S. Shaikh

Widener University, Chester, PA 19013

sohail.shaikh@widener.edu

1. Abstract store and share multimedia documents inter-

Distributed, networked multimedia informa-
tion systems will be a critical component of
technology-based information infra-structures
in the future. Several ground breaking appli-
cations have already appeared, and more are
many are expected to follow. Innovations in
hardware and software are feeding this revo-
lution. In this paper, the notion of QOP for
multimedia data transferred over the networks
actively. In another application area known
as tele-medicine, Broadband Integrated Ser-
vices Digital Network (B-ISDN) using giga-
bit technology will allow the development of
medical communication systems capable of de-
livering medical services to distant communi-
ties as well as remote access to patient data.
In the news-media industry, several joint ven-
tures aimed at developing interactive multi-
media news (e.g., news-on-demand, personal-
is developed. Multimedia data synchronization ized multimedia news ltering etc.) have been
requirements are studied, and several synchro- formed in recent months. Plans also include to
nization protocols are presented. We discuss provide services such as tele-shopping, consult-
the management of pre-orchestrated as well as ing and other convenient and nancially viable
live multimedia data. It is anticipated that home information services.
with proper research breakthroughs, broad- This paper is organized as follows: Section 3
band multimedia networking technologies will introduces the issues in scheduling for synchro-
bring about spectacular changes in the ways nized delivery of multimedia data. Models
that we store, process, and use information. and data abstractions commonly used to spec-
ify the temporal relationships, along with the
2. Introduction presentation quality, of multimedia objects are
also discussed. In Section 4 various synchro-
nization mechanism for multimedia informa-
Networked and distributed multimedia in- tion are discussed. Finally, Section 6 concludes
formation systems will play an important role the issues related to multimedia communica-
in engineering and technological progress in the tion synchronization.
future. They are catalysts for new research in
a number of areas including ecient data in- 3. Multimedia Synchronization
put devices, high bandwidth digital networks,
storage systems with extremely large capaci- and Quality Considerations
ties, data modeling of image and video data,
very large distributed databases, real-time op- The applications mentioned in the previous
erating systems, computer graphics and anima- section provide a synopsis of a broad class of
tions, and human-computer interaction. With distributed multimedia services that fall un-
developments in such areas rapidly becoming der the general framework of Computer Sup-
a reality, many ambitious multimedia based ported
The Collaborative
development ofWork (CSCW). Figure
the architecture in Fig-1
projects are being pursued by the the indus- depicts
ure 1, over broadband networks poses new chal-a
the functional architecture of such
try, academia, and the governments. Digital system andstorage
lenges for emphasizes the important diverse
and communication role of
multimedia libraries, one of the projects cur- broadband networking technologies in support-
multimedia objects like video, audio, images,
rently being supported by the U.S. govern- ing
etc. these applications.
Multimedia application may require live
ment, will promote aordable remote learn- data being generated in real time at remote lo-
ing environments and allow users to author, cations, and/or some form of pre-orchestrated
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.
 Application Collaboration Layer
System Management Layer
Multiplicity Communication Layer
Broadband Multimedia Network Layer
exibility in allocating re-
information (or documents) stored at vari-
ous servers interconnected over broadband net-
works [5, 8, 10]. Retrieval and communica-
tion of multimedia data imposes diverse per-
formance and reliability characteristics, on the
underlying communication infrastructure. For
example, while transmitting video and audio
data streams, network delays must be bounded
to maintain inter-stream and intra-stream tem-
poral synchronization requirements [8]. Inter-
stream synchronization deals with the syn-
chronized play-out of related streams, whereas
intra-stream is required for the continuous pre-
sentation of each stream
Temporal synchronization requires that the
system must observe some time relationships
(constraints) among various data objects in or-
der to present the information to the user in
a meaningful way. These relationships can be
natural or synthetically created [7]. Simulta-
neous recording of voice and video through a
VCR, is an example of natural relationship be-
used for data compression, the distribution of
data, and random communication delays intro-
duced by the network. Such factors make the
provision of these capabilities infeasible with
the current technologies.
Allocation of capacity and buer need to
be consistent with the synchronous playback
of multimedia information at the destination.
Knowledge of the temporal characteristics and
their inter-relationship of multimedia objects
gives considerable
sources for managing multimedia trac [6].
By identifying the temporal inter-relationships
and the desired presentation quality of the mul-
timedia information, the overall resource re-
quirements can be established precisely. In this
paper, we elaborate on these issues and focus
our discussion on synchronization and resource
management schemes in broadband networks.
Specially we focus on a set of Quality of Pre-
sentation (QOP ) parameters, that specify the
desired quality of presentation of multimedia
information and are assumed to be given by
users as part of their multimedia information
attributes. Subsequently, we discuss the role of
QOP parameters in designing various network
management schemes for resource allocation.
Many existing networks assume a packet
switched network architecture, like that in the
asynchronous transfer mode (ATM) network
environment. Due to resource constraints the
system may not be able to provide high quality
services. For a qualitative evaluation of a ser-
tween audio and video information. A voice vice, a set of QOP parameters can be dened
annotated slide show, on the other hand, is to specify the desired quality of presentation of
an example of synthetically created relationship multimedia information, and are assumed to be
between audio and image information. In this
case, change of an image and the end of its
verbal annotation, represent a synchronization
point in time.
A user can randomly access various objects,
while browsing through a multimedia informa-
tion system. In addition to simple forward
play-out of time-dependent data sequences,
other modes of data presentation are also vi-
able, and should be supported by a multime-
dia information management system. These
include user-interactions like reverse play-out,
fast-forward/fast-backward play-out, and ran-
dom access of arbitrarily chosen segments of
the presentation. These operations are quite
common in TV technology, (e.g. VCRs), but
their implementation in distributed multime-
dia systems is a challenging task. This is due
to the non-sequential storage of multimedia ob-
jects, the diversity in the features of hardware
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.
specied by users. For example, one or more
of these parameters may be used to quantify
the percentage of multimedia data that can be
dropped by the server, in case the resources
are limited. Restricted buering capability or
limited network capacity may also require the
dropping of some data and delivering partial
multimedia streams. Loss of data for a media
object can aect the presentation rate or reso-
lution of that object. Generally, the acceptable
bound on this loss of data can be expressed
in terms of some maximum allowable percent-
age data loss. It is possible that a network
may transmit several objects concurrently over
independent virtual channels having dierent
delay characteristics. Therefore these objects
can experience dierent jitter delays, leading to
inter-stream synchronization failure as a result
of missing presentation deadline. Objects with
missed deadlines start lagging while the data
 on other streams is continuously used up dur-
ing presentation. The acceptable delay or slip-
page of a media object (video clip, an image,
audio segment, etc.) with respect to its dead-
line, depends on object type and the multime-
To support synchronized delivery of diverse
multimedia objects with dierent QOP re-
quirements, the networks of today face the fol-
lowing challenges;
dia application and can be specied as maxi- - Allocation of resources for optimum uti-
mum tolerable percentage of deadline misses. lization.
- Transport of each multimedia stream
4. Synchronization Protocols over
Broadband Networks
Presentation of pre-orchestrated or live mul-
timedia information requires synchronous play-
out of time-dependent multimedia data accord-
within the specied QOP parameters.
- Simple inter-switch signaling for low con-
nection setup latency.
We now elaborate on various synchronization
mechanisms for delivery of multimedia infor-
mation over broadband networks.
ing to some specied temporal relations. At
the time of creation of multimedia informa-
tion, the temporal constraints among various
data streams, necessary at the time of play-
back, are expressed in the form of a model [7].
Anisochronous data (e.g text and images) need
to be available to the play-out devices at the
destination prior to their play-out deadlines.
On the other hand isochronous objects, such
as video and audio, can be transmitted at the
same rate as the play-out rate [4]. To mini-
mize destination buer requirements and pro-
vide continuity in playback, the rates of com-
munication and presentation of an isochronous
object need to be equal. Assuming a packet
switched network, the transmission of individ-
ual units of information need to be intelligently
controlled. For this purpose media objects can
be divided into smaller units of information
which are controlled by the application process.
We refer to such a unit as Synchronization In-
terval Unit (SIU). For example, in a video clip
such a unit can be taken as 301 th of a second,
which corresponds to the play-out duration of a
single frame. In other words, an SIU for a video
object could be a video frame. For audio data,
the SIU can be an audio sample. In case of
discrete media objects (images and text), the
entire object can be viewed as an SIU. Since
SIUs are the basic units for play-out devices
as well as the units to ensure synchronization,
their delayed arrivals may make them obsolete
in the play-out process. Therefore, it is impor-
tant that the presentation deadlines of SIUs
should be known a-priori. These deadlines can
be easily derived from the object's deadline it-
self. All SIUs associated with the same tem-
poral interval must be played out simultane-
ously within their overlapping intervals, which
in turn requires the source to deliver SIUs for
all objects at a constant rate.
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.
5. Server Based Scheduling
To ensure the availability of each SIU at the
destination before the play-out deadline in the
presence of random network delays, the SIUs
need to be scheduled ahead of their respective
deadlines by a factor greater than the maxi-
mum expected delay. This pre-scheduling time
is referred to as the control time. Since the
delays are random, some SIUs may arrive ear-
lier than their deadlines, and will need to be
buered. The amount of buering required de-
pends on the control time, which in turn is
dependent on the network delay distributions.
In case of multiple streams, inter-stream syn-
chronization along with intra-stream is also re-
quired.
In server-based scheduling scheme, the net-
work can use a static reservation scheme, and
can provide multiple channels with guaranteed
bandwidth and delay bounds. Each channel
may have dierent delay bounds and band-
width. Since the capacity of each channel and
their number may not be sucient for transfer-
ring multimedia data within the given QOP re-
quirements, the connection request may be de-
nied. Alternatively, some amount of multime-
dia data can be pre-fetched at the user-station
for presentation later at the play-out deadline.
The pre-fetched data needs to be buered prior
to its synchronized play-out. This results in
some initial delay which increases the system
response time. The objective of the resource
management scheme is to reserve resources to
minimize the system response time and desti-
nation buer requirements, while ensuring syn-
chronized play-out of multimedia data. Since
the presentation process has to be delayed be-
cause of the resource constraints, this schedul-
 ing scheme is more suited for application re-
quiring pre-orchestrated or stored multimedia
data rather than live data.
For a multi-channel connection, the schedul-
ing of n SIUs on m channels is equivalent to as-
signing n independent tasks to m uniform par-
estimate of the time to asynchrony of each
stream, can adaptively control the feedback
transmission rate from that remote play-out
device [9]. The control is exercised to mini-
mize the associated overheads without allowing
the asynchrony to exceed the tolerance limits
allel processors with dierent processing speeds
which is a well known NP-hard problem [3].
Thus polynomial time algorithms for optimal
solutions are unlikely to exist. To service user
requests the server needs to schedule the multi-
media documents in real-time. Various heuris-
tic greedy algorithms can be used to solve this
problem [1].
To ensure continuous and synchronized
play-out of multimedia data across the remote
user stations, in the absence of globally syn-
chronized clocks, a feedback technique, sug-
gested by Ramanathan et al. [9] can be used.
In this approach light-weight feedback mes-
sages (referred to as feedback units), transmit-
ted periodically by the remote user stations
back to the multimedia server, are used by the
server to estimate the play-out instants of me-
dia streams at the respective remote station.
The server uses these estimates to detect im-
pending buer over
ow or under
ow at the sive policy
uctuates at lower asynchrony lev-
remote sites. Based on these it can readjust
the transmission rate of the multimedia data
streams so as to avoid the loss of data and
synchronization at the remote station. Given
buer sizes available at the remote stations the
minimum rate at which feedback units must
be transmitted for maintaining play-out con-
tinuity is determined. Upon receiving a feed-
back unit, the multimedia server estimates the
earliest and latest playback times of the SIUs
which are yet to be transmitted to the re-
mote users. From this information the earli-
est and latest possible transmission times are
computed. The transmission of SIUs at the
earliest transmission times result in high des-
tination buer occupancy, while transmission
at the latest transmission times keep the des-
tination buer usage at a minimum. Frequent
transmission of feedback units, i.e. a high feed-
back ratio, enables the server to make more
precise estimates of transmission times for bet-
ter approximation to the actual playback rate.
However this approach places additional load
on the remote stations, network, and multime-
dia server which have to generate, carry, and
compute the feedback units and transmission
times, respectively.
Re-synchronization strategies by which the
multimedia server, based on its most recent
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.
specied by the QOP parameter \". The re-
synchronization policies are:
- conservative: This scheme reacts only
when play-out of dierent streams is guar-
anteed to be asynchronous
- aggressive: In this approach server re-
acts as soon as there is even a slight chance
that playback is asynchronous
- probabilistic: Server reacts on the av-
erage (assuming the network delay distri-
butions and play-out rate variations are
known)
By applying these re-synchronization strate-
gies to video and audio playback it has been
shown [9] that the conservative policy performs
well at lower levels of asynchrony, but its eec-
tiveness reduces at higher levels of asynchrony.
On the other hand, the performance of aggres-
els, but out performs the conservative strategy
at higher levels. Both the policies become in-
eective in a high network jitter environment.
In this case the probabilistic policy continues
to perform uniformly well.
Synchronized delivery of multimedia data
can also be ensured via intelligently control-
ling network resources. The crux of the ap-
proach is to guarantee availability of resources
on-demand or in static mode, so that media
streams are not constrained at the time of
transmission. Recently new concepts of re-
source controlled synchronization of multime-
dia services have emerged. In this section we
elaborate on these approaches with special em-
phasis on broadband land-based packet switch-
ing and mobile networks.
In a network, trac load at each switch
can change dynamically due to various factors,
such as the number of users concurrently served
by that switch, the changing level of concur-
rency of multimedia data streams, and the ini-
tiation of new requests. In the network shown
in Figure 2, if Virtual Path (VP) 1 and VP 3
are used by multimedia server 1 and 2 for trans-
mitting data to Users 2 and 1 respectively, then
 the switch Sw2 has to manage multimedia data A fair capacity allocation policy at a switch
streams for both users. requires that if transmission of streams need
to be degraded, then degradation should be
evenly spread across all the streams that are
being transmitted concurrently.
Multimedia server 3 Under such
a policy, the problem of nding dropping ra-
tios for all concurrent objects, in an inter-
val, to conform to the limited switch capacity,
can be formulated as an optimization problem
with constraints specied by the QOP param-
Multimedia server 1 eters [2].
Sw 2 Such optimization
Sw 3 can be integrated as a
part of the overall resource reservation and al-
location protocol. The connection User 1
request at
Sw 1 the kth switch, contains the synchronization,
throughput and reliability requirements for the
Sw 4 request. If the switch
Sw 5 can serve a the requested
connection while satisfying the specied relia-
bilities of existing ones for theUserduration
2 of the
connections, the new connection is accepted
and resources assigned accordingly. On the
other hand, if an optimization solution is not
feasible, then the request for new connection
is blocked and a request-denied signal is sent
Multimedia server 2 to the previous switch on the VP. The switch,
Sw(k,1) , checks if it alsoVPis1 on the next VP, i.e.
V P(j+1) . If yes, then VP it 2modies the OCPN
for the new delays (if VP they3 are dierent) and
repeats the allocation procedure. In case the
switch is not on the (j + 1)th VP, then the
switch sends the request-denied signal to the
In static resource allocation schemes, xed previous switch, i.e. Sw(k,2) .
bandwidth and buer can be allocated at each
switch. This assignment can be done at the 6. Conclusion
time of VP establishment. Although such
schemes are simple, they can be very inecient
and wasteful for managing multimedia con- In this paper we have discussed issues re-
nections when bandwidth requirements may lated to synchronization of multimedia infor-
vary signicantly for the duration of a connec-
tion. On the other hand dynamic resource al-
location schemes, while being able to allocate
capacity/buer eciently, may require elabo-
rate inter-switch signaling. This may result in
longer connection setup delays. Furthermore,
the network may not allow dynamic manage-
ment or each switch may not be capable of dy-
mation over broadband networks. Various as-
pects of presentation quality and their im-
pact on system resources form the basis of
designing multimedia synchronization proto-
cols for supporting distributed multimedia ser-
vices. For a multimedia document, the tem-
poral relationships among multimedia objects
and their quality characteristics need to be
namically assigning resources for each channel specied. Various specication models used
at any arbitrary instant of time. The devel- for dening the characteristics of the multime-
opment of high-speed broadband networks has dia objects, specically for transport over com-
greatly enhanced the bandwidth and process- munication networks, are introduced in Sec-
ing capability available at each switch in a net- tion 3. Presentation of time dependent mul-
work. Also, the usual duration of multimedia timedia information involves intra-stream and
connections is considerably larger than the ini- inter-stream synchronization. Random delays
tial connection establishment delay. Thus dy- in a communication network can result in loss
namic resource allocation schemes can be used of intra-stream synchronization failure. Mul-
for optimum network resource utilization. tiple streams may be transported over dier-
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.
 ent channels, having dierent delay character-
istics, resulting in inter-stream synchronization
failure. Dierent schemes and techniques to re-
cover from intra-stream and inter-stream asyn-
chrony are outlined in Section 4. In addition,
allocation and management of constrained net-
work resources for synchronized delivery of
multimedia information within the quality con-
[10] M. Woo, N. U. Qazi, and A. Ghafoor. A
synchronization framework for communica-
tion of pre-orchestrated multimedia informa-
tion. IEEE Network Magazine, 8(1):52{61,
January/ February 1994.
straints specied in the model, are also de-
scribed. Depending on the type of multime-
dia service and application, each technique has
certain advantages and disadvantages. Cur-
rently, synchronization of multimedia informa-
tion over broadband land-based and mobile
networks is an active area of research with con-
stant innovations. With all this activity, it is
expected that new and improved synchroniza-
tion techniques will revolutionize the area of
distributed multimedia systems.
References
[1] S. Baqai, M. F. Khan, M. Woo, S. Shinkai,
A. Khokhar, and A. Ghafoor. Quality-
based evaluation of multimedia synchroniza-
tion protocols for distributed multimedia in-
formation systems. IEEE Journal on Selected
Areas in Communications, 14(7):1388{1403,
September 1996.
[2] S. Baqai, M. Woo, and A. Ghafoor. Net-
work resource management for enterprise-
wide multimedia services. IEEE Communi-
cations, 34(1):78{83, January 1996.
[3] J. Blazewicz, K. Ecker, G. Schmidt, and
J. Weglarz. Scheduling in Computer and
Manufacturing Systems. Springer-Verlag,
Berlin
Heidelberg, 1993.
[4] D. Ferrari. Client requirements for real-time
communication services. IEEE Communica-
tion Magazine, 28(11):65{72, 1990.
[5] B. Furht. Multimedia systems: An overview.
IEEE Multimedia, 1(1):47{59, Spring 1994.
[6] A. Lazar and G. Pacici. Control of resources
in broadband networks with quality of ser-
vice guarantees. IEEE Communications Mag-
azine, 29(10):66{73, October 1991.
[7] T. Little and A. Ghafoor. Synchronization
and storage models for multimedia objects.
IEEE Journal on Selected Areas in Commu-
nications, 8(3):413{427, April 1990.
[8] T. Little and A. Ghafoor. Multimedia syn-
chronization protocols for broadband inte-
grated services. IEEE Journal on Selected Ar-
eas in Communications, 9(9):1368{1382, De-
cember 1991.
[9] S. Ramanathan and P. V. Rangan. Adap-
tive feedback techniques for synchronized
multimedia retrieval over integrated net-
works. IEEE/ACM Transactions on Net-
working, 1(2):246{259, April 1993.
Authorized licensed use limited to: Sree Chitra Thirunal College of Engineering. Downloaded on June 10,2010 at 11:53:02 UTC from IEEE Xplore. Restrictions apply.