Multimedia

IT342
Authoring
Chapter 1
1
Outlines
• What is Multimedia?
• Components of Multimedia
• Multimedia Research Topics and Projects
• Multimedia and Hypermedia

IT342
• Multimedia Authoring metaphors
• Multimedia Production
• Multimedia Presentation
• Some Technical Design Issues
• Automatic Authoring

2
 What is Multimedia?
• When different people mention the term multimedia, they often
have quite different, or even opposing, viewpoints.
A PC vendor:
• a PC that has sound capability, a DVD-ROM drive, and perhaps the superiority of
multimedia-enabled microprocessors that understand additional multimedia
instructions.

IT342
A consumer entertainment vendor:
• interactive cable TV with hundreds of digital channels available, or a cable TV-like
service delivered over a high-speed Internet connection.

A Computer Science (CS) student:

• applications that use multiple modalities, including text, images, drawings
(graphics), animation, video, sound including speech, and interactivity.

• Multimedia and Computer Science: 3

• – Graphics, HCI, visualization, computer vision, data compression, graph
theory, networking, database systems. Multimedia and Hypermedia
What is Multimedia?
A good general definition is:
• Multimedia is a combination of text, graphic, sound,
animation, and video that is delivered interactively to the
user by electronic or digitally manipulated

IT342
4
Why to use Multimedia?
• According to Dr. Albert Mehrabian, a specialist
in interpersonal communication at the University
of California, says:
People recall 20% of what they see

IT342
40% of what they see & hear
70% of what they see, hear & do

Multimedia allows for seeing, hearing and doing

5
Components of Multimedia

Text

IT342
Animation Images
Multimedia
Components

Video Audio
6
Components of Multimedia
• Multimedia involves multiple modalities of text, audio, images,
drawings, animation, video and Interactivity .
• Examples of how these modalities are put to use:

IT342
Video Distributed Tele-medicine Making “Augmented”
teleconferencing lectures for multimedia reality: placing
higher education components real-appearing
editable computer
graphics and
video objects into
scenes. ex

Using voice- Searching in

recognition to (very) large video 7
build an and image
interactive databases for
environment target visual
objects
Multimedia Research Topics
and Projects
• To the computer science researcher, multimedia consists of a
wide variety of topics:
1. Multimedia processing and coding: multimedia content analysis,
content-based multimedia retrieval, multimedia security,
audio/image/video processing, compression, etc.

IT342
2. Multimedia system support and networking: network protocols,
Internet, operating systems, servers and clients, quality of service
(QoS), and databases.
3. Multimedia tools, end-systems and applications: hypermedia
systems, user interfaces, authoring systems.
4. Multi-modal interaction and integration: “ubiquity” — web-
everywhere devices, multimedia education including Computer
Supported Collaborative Learning, and design and applications of 8
virtual environments.
Current Multimedia Projects
• Many exciting research projects are currently underway. Here are a few of
them:

Camera-based object tracking technology:

• Tracking of the control objects provides user control of the process.

IT342
3D motion capture:
• Used for multiple actor capture so that multiple real actors in a virtual studio can
be used to automatically produce realistic animated models with natural
movement.

Multiple views:
• Allowing photo-realistic (video-quality) synthesis of virtual actors from several
cameras or from a single camera under differing lighting.

Specific multimedia applications: 9

• Aimed at handicapped persons with low vision capability and the elderly — a rich
field of endeavor.
Current Multimedia Projects
• Many exciting research projects are currently underway. Here are a few of
them:

Digital fashion:
• Aims to develop smart clothing that can communicate with other such enhanced
clothing using wireless communication, so as to artificially enhance human
interaction in a social setting.

IT342
Electronic House call system:
• An initiative for providing interactive health monitoring services to patients in
their homes

Augmented Interaction applications:

• Used to develop interfaces between real and virtual humans for tasks such as
augmented storytelling.
10
 Multimedia and Hypermedia
• History of Multimedia:
Newspaper:
• perhaps the first mass communication medium, uses text, graphics, and images.

Motion pictures:

IT342
• conceived of in 1830’s in order to observe motion too rapid for perception by the
human eye.

Wireless radio transmission:

• Guglielmo Marconi, at Pontecchio, Italy, in 1895.

Television:
• the new medium for the 20th century, established video as a commonly
available medium and has since changed the world of mass communications.

The connection between computers and ideas about multimedia covers what 11
is actually only a short period
Hypermedia and Multimedia
• A hypertext system: meant to be read nonlinearly, by
following links that point to other parts of the document, or to
other documents (Fig. 1.1)
• HyperMedia: not constrained to be text-based, can include
other media, e.g., graphics, images, and especially the

IT342
continuous media – sound and video.
• The World Wide Web (WWW) — the best example of a
hypermedia application.

• Multimedia means that computer information can be

represented through audio, graphics, images, video, and 12
animation in addition to traditional media.
Hypermedia and Multimedia

IT342
13
 IT342
• Fig 1.1: Hypertext is nonlinear 14
• Examples of typical present multimedia applications include:

• Digital video editing and production systems.

• Electronic newspapers/magazines.
• World Wide Web.
• On-line reference works: e.g. encyclopedia, games, etc.

IT342
• Home shopping.
• Interactive TV.
• Multimedia courseware.
• Video conferencing.
• Video-on-demand.
• Interactive movies.

15
Multimedia Software Tools
• Music Sequencing and Notation – Cakewalk, Cubase,
Macromedia Soundedit
• Digital Audio – Cool Edit, Sound Forge, Pro Tools
• Graphics and Image Editing – Adobe Illustrator, Adobe

IT342
Photoshop, Macromedia Fireworks, Macromedia Freehand
• Video Editing – Adobe Premiere, Adobe After Effects, Final Cut
Pro
• Animation – Multimedia APIs: Java3D, DirectX, OpenGL –
Rendering Tools: 3D Studio Max, Softimage XSI, Maya,
RenderMan
• Multimedia Authoring – Macromedia Flash, Macromedia
Director, Authorware, Quest 16
 Multimedia Authoring
• Authoring is the process of creating multimedia applications
• Metaphors are methodologies employed by the authoring
system to accomplish its task (create multimedia
applications).

IT342
• Some common authoring metaphors are:
1. Scripting language metaphor
2. Slide show metaphor
3. Hierarchical metaphor
4. Iconic/flow-control metaphor
5. Frames metaphor
6. Card/scripting metaphor
17
7. Cast/score/scripting metaphor
Multimedia Authoring Metaphors
1. Scripting Language Metaphor: use a special language to
enable interactivity (buttons, mouse, etc.), and to allow
conditionals, jumps, loops, functions/macros etc.

IT342
-- load an MPEG file
extFileName of MediaPlayer "theMpegPath" =
"c:\windows\media\home33.mpg";
-- play
extPlayCount of MediaPlayer "theMpegPath" = 1;
-- put the MediaPlayer in frames mode (not time mode)
extDisplayMode of MediaPlayer "theMpegPath" = 1;
-- if want to start and end at specific frames:
extSelectionStart of MediaPlayer "theMpegPath" = 103;
extSelectionEnd of MediaPlayer "theMpegPath" = 1997;
-- start playback
get extPlay() of MediaPlayer "theMpegPath";
18
Multimedia Authoring Metaphors
2. Slide Show Metaphor: A linear presentation by default,
although tools exist to perform jumps in slide shows, EX.
PowerPoint or ImageQ.

IT342
2. Hierarchical Metaphor: User-controllable elements are
organized into a tree structure — often used in menu-driven
applications.
19
Multimedia Authoring Metaphors
4. Iconic/Flow-control
Metaphor: Graphical icons
are available in a toolbox,
and authoring proceeds by
creating a flow chart with

IT342
icons attached (Fig. 2.1),
EX. Authorware by
Macromedia. As well as
simple flowchart elements,
such as an IF statement, a
CASE statement, group of
elements using MAP
(subroutine). Also, simple
animation is possible. 20
Multimedia Authoring Metaphors
5. Frames Metaphor: Like
Iconic/Flow-control
Metaphor; however links
between icons are more
conceptual, rather than
representing the actual

IT342
flow of the program (Fig.
2.2), EX. Quest by Allen
Communication.
Flowchart consists of
modules composed of
frames. Frames are
constructed from objects,
such as text, graphics,
audio, animations and
video, all of which can 21
respond to events.
Multimedia Authoring Metaphors
6. Card/Scripting Metaphor: Uses a simple index-card
structure — easy route to producing applications that use
hypertext or hypermedia; used in schools.
e.g., SuperCard, HyperCard by Apple

IT342
22
Multimedia Authoring Metaphors
7. Cast/Score/Scripting Metaphor:
• Time is shown horizontally; like a spreadsheet: rows,
or tracks, represent instantiations of characters in a
multimedia production.

IT342
• Multimedia elements are drawn from a cast of
characters, and scripts are basically event-procedures
or procedures that are triggered by timer events.
• Director, by Macromedia, is the chief example of this
metaphor. Director uses the Lingo scripting language,
an object-oriented event-driven language.
23
Multimedia Authoring Metaphors
7. Cast/Score/Scripting Metaphor:

Cast

IT342
Score

24
 Multimedia Production
• People produce multimedia involve an art director, graphic designer,
production artist, producer, project manager, writer, user interface
designer, sound designer, videographer, and 3D and 2D animators, as
well as programmer.
• Multimedia production design phase consists of storyboarding,

IT342
flowcharting, prototyping and user testing as well as a parallel
production of media.
• A storyboard depicts the initial idea content of a multimedia
concepts in a series of sketches (keyframes).
• A flowchart organizes the storyboards by inserting navigation
information (multimedia concept’s structure and user interaction).
• A prototype includes development of detailed functional
specification (walk-through, screen action, user interface).
25
• User testing is extremely important before final development phase
(tools may be used).
 Multimedia Presentation
• Features that affect presenting multimedia content :
1. Graphics Styles: Human visual dynamics are considered in
regard to how such presentations must be constructed.
1. Color principles and guidelines: Some color schemes and art styles are
best combined with a certain theme or style. A general hint is to not use

IT342
too many colors, as this can be distracting. A color contrast program
can be used based on the concept: If the text color is some
triple(R,G,B), a legible color for the background is that color subtracted
from the maximum (here assuming max=1):
(R, G, B)  (1 − R, 1 − G, 1 − B)
This is called the principal complementary color, for color values in the
range 0 to 1
2. Fonts: For effective visual communication in a presentation, it is best to
use large fonts (i.e., 18 to 36 points), and no more than 6 to 8 lines per
26
screen (fewer than on this screen!).
 Li & Drew
• Fig. 2.4: Colours and fonts [from Ron Vetter].
27
 IT342
• Fig. 2.5: Program to investigate colours and
readability. 28
 IT342
Fig. 2.6
29
shows a “colour wheel”, with opposite colours equal to (1-R, 1-G, 1-B)
Multimedia Presentation
2. Sprite Animation: “sprites are special graphic objects that
can move independently of the screen background”.
based on, Suppose we have an animation figure, as in Fig. 1 (a).
Now create a 1-bit mask M, as in Fig. 1 (b), black on white, and
accompanying sprite S, as in Fig. 1 (c).

IT342
Fig. 1: Sprite creation: Original, mask image M, and sprite S (“Duke”
figure courtesy of Sun Microsystems.) 30
Multimedia Presentation
• We can overlay the sprite on a colored background B, as in Fig. 2
(a) by first ANDing B and M, and then ORing the result with S,
with final result as in Fig. 2 (e).

IT342
Fig. 2: Sprite animation: (a): Background B. (b): Mask M. (c): B AND M. (d): 31
Sprite S. (e): B AND M OR S
Multimedia Presentation

IT342
32
 IT342
33
Multimedia Presentation
3. Video transitions: is an effective way to indicate a
change to the next section. It is a semantic means to
signal “scene changes” and often carry semantic
meaning. Many different types of transitions:

IT342
• Cut: an abrupt change of image contents formed by abutting two
video frames consecutively. This is the simplest and most
frequently used video transition.

34
Multimedia Presentation
• Wipe: a replacement of the pixels in a region of the viewport
with those from another video. Wipes can be left-to-right,
right-to-left, vertical, horizontal, like an iris opening, swept out
like the hands of a clock, etc.

IT342
35
Multimedia Presentation
• Dissolve: replaces every pixel with a mixture over time
of the two videos, gradually replacing the first by the
second. Most dissolves can be classified as two types:
• Cross dissolve (Type I)

IT342
• Dither dissolve (Type II)

36
Multimedia Presentation
• Cross dissolve (Type I):
Blends the first clip into the second clip. Every pixel is affected
gradually.

IT342
37
Multimedia Presentation
• Dither dissolve (Type II)
• Determined by α(t), increasingly more and more pixels in video A
will abruptly (instead of gradually as in Type I) change to video B.

IT342
38
 Multimedia Presentation
• Fade-in and fade-out are special types of Type I dissolve:
video A or B is black (or white).

• Wipes are special forms of Type II dissolve in which changing

pixels follow a particular geometric pattern.

IT342
39

http://www.mediacollege.com/video/editing/transition/types.html
Some Technical Design Issues
• Computer Platform: Much software is ostensibly “portable” but
cross-platform software relies on run-time modules which may not
work well across systems.
• Video format and resolution: The most popular video formats —
NTSC, PAL, and SECAM— are not compatible, so a conversion is

IT342
required before a video can be played on a player supporting a
different format.
• Memory and Disk Space Requirement: At least 128 MB of RAM and
20 GB of hard-disk space should be available for acceptable
performance and storage for multimedia programs.
• Delivery Methods: CD-ROMs: may be not enough storage to hold a
multimedia presentation. As well, access time for CD-ROM drives is
longer than for hard-disk drives. Electronic delivery is an option, but
depends on network bandwidth at the user side (and at server). A 40
streaming option may be available, depending on the presentation.
Automatic Authoring
• Facilitating automatic authoring is either by creating new
multimedia presentations or by automatic creation of more
useful multimedia documents from existing resources.
• Hypermedia documents: Generally, three steps are to
produce documents to be viewed nonlinearly:

IT342
1. Capture of media: From text or using an audio digitizer or video
frame-grabber; is highly developed and well automated.
2. Authoring: How best to structure the data in order to
support multiple views of the available data, rather than a
single, static view.
3. Publication: i.e. Presentation, is the objective of the
multimedia tools we have been considering. 41
Automatic Authoring
• Externalization versus linearization:
• (a) the essential problem involved in communicating ideas without
using a hypermedia mechanism (i.e., linearization).

IT342
42
• (b) In contrast,
hyperlinks allow us the freedom to partially mimic the author’s
thought process (i.e., externalization).

IT342
43
• Using, e.g., Microsoft Word, creates a hypertext version of a
document by following the layout already set up in chapters,
headings, and so on.
• But problems arise when we actually need to automatically extract
semantic content and find links and anchors (even considering just

IT342
text and not images etc.) Fig. 2.13 displays the problem.

44
 IT342
Fig. 2.13: Complex information space [from David Lowe].

• Once a dataset becomes large we should employ database

methods. The issues become focused on scalability (to a large
dataset), maintainability, addition of material, and reusability.

45
Automatic Authoring
• Semi-automatic migration of hypertext
• The structure of hyperlinks for text information is simple: “nodes”
represent semantic information and these are “anchors” for links to
other pages.

IT342
Fig. 2.14: Nodes and anchors in hypertext [from David Lowe]. 46
Automatic Authoring
• Hyperimages
• We need an automated method to help us produce true hypermedia:

IT342
Fig. 2.15: Structure of hypermedia [from David Lowe]. 47
• Can manually delineate syntactic image elements by masking image
areas.
• Fig. 2.16 shows a “hyperimage”, with image areas identified and
automatically linked to other parts of a document:

IT342
Fig. 2.16: Hyperimage [from David Lowe].
48
Summary
• Automatic Authoring: an advanced helper for creating new
multimedia presentations or a mechanism to facilitate
automatic creation of more useful multimedia documents
from existing sources

IT342
• Multimedia Authoring Metaphors: for easier understanding
of the methodology employed to create multimedia
applications.

• Multimedia Production: how to produce multimedia

• Multimedia Presentation: effects for presenting multimedia

content as well as guidelines for content design 49