Android Multimedia Overview

February 11th 2009

David Gottardo, System SW Architecture at Texas Instruments France
Email: d-gottardo@ti.com

Android Architecture
Table of Contents

Audio Flinger: Audio manager
Surface Flinger: Surface manager /
composite
Multimedia Framework
MM Playback
(MM Record)

Android Components
Red border is corresponding to components not supported by TI SW foundation (Modem vendors, etc.)
W
L
A
N
G
P
S
U
S
B
B
T
V
i
b
r
a
t
o
r

R
-
T
o
u
c
h
N
A
N
D
M
M
C
/
S
D
K
e
y
b
o
a
r
d
C
a
m
e
r
a
D
i
s
p
l
a
y
T
V

O
u
t
Peripherals
Peripherals
Core HW Platform OMAP 3630
ARM Cortex A8
TWL 5030
S
D
R
A
M
Linux Kernels
Bluetooth
Driver
Camera
Driver
Power
Management
WiFi Driver
Keypad
Driver
USB Driver
Sensors
Driver
Led Driver GPS Driver
Flashlight
Driver
Android Hardware Abstraction Layer
Graphics Audio Camera Bluetooth WiFi Flashlight GPS Led Sensors
Mount
Service
Power
Alarm & RTC Logging
Telephony
Radio
Layer
Battery
3430 Machine
Support
MM ARM
Codecs
MM DSP
Codecs
Applications
L
E
D
Touch Driver
C
-
T
o
u
c
h
IVA 2+ DSP
MM ARM
Codecs
MM DSP
Codecs
Nand
support
B
a
c
k
l
i
g
h
t
F
M

R
a
d
i
o
U
A
R
T
UART
Driver
Backlight
Driver
FM Radio
Driver
Audio Driver
NL5500 WL1271
M
M

A
u
d
i
o
B
a
t
t
e
r
yF
l
a
s
h
S
P
I
SPI Driver
Libraries
Surface Flinger OpenGL/ES
SGL
Media
Framework
FreeType
SSL
SQLite WbKit
Libc
Android Runtime
Dalvik Virtual
Machine
Core Library
Vibrator
Home
Contacs
Dailer
Voice Dail
SMS/MMS
Email
IM
Calendar
Browser
Media Player
Camera
Albums
Alarm
Clock
Calculator
Etc.
C64x DSP
Binder (IPC)
Driver
Application Framework
Content
Providers
View
System
Resource
Manager
Location
Manager
Etc.
RIL Library
System Services
Activity
Manager
Window
Manager
Package
Manager
Notification
Manager
Telephony
Manager
App
launcher
Status Bar
GFX (SGX)
Driver
Display
Driver
Audio Flinger
Media
Android Surface Manager
Client/Server Architecture
Android graphics system is based on a Client/Server Architecture
Server side: Surface Flinger is a service coded in C++
Manages access to display subsystem
Seamlessly composites 2D and 3D graphic layers from multiple
applications
Client side: applications can draw on surfaces using two paths:
Use widgets/View/Canvas provided by the JAVA APIs and managed
by C++ external Skia library
Access OpenGL through JNI to directly create 3D surfaces
OpenGL
Java Native
Interface
2D Apps / 3D Apps
Java View/
Widget/
Canvas
Surface
SKIA (SGL)
SurfaceFlinger
Server side native library (C++):
frameworks/base/libs/surfaceflinger/
Provides system-wide surface composer, handling all surface
rendering to frame buffer device.
Can combine 2D and 3D surfaces (Z-buffer) from multiple
Applications
Can use OpenGL ES and 2D hardware accelerator for its
compositions.
Double-buffering using page-flip.
Surfaces passed as buffers via Binder IPC calls (Androids interprocess
communication pipe).
SurfaceFlinger
(libsurfaceflinger)
Frame
Buffer
3D Surface 2D Surface 3D Surface
OpenGL ES (HW or SW)
Surface Management
Android apps are built using View and ViewGroup objects:
Basic UI building blocks (text, buttons, checkboxes, etc)
Responsible for handling drawing, events, measuing.
Each View is tied to a Surface, which is double buffered.
Surfaces are managed as follows:
ViewRoot (app window) asks SurfaceFlinger for Canvas for View to draw on.
View draws its data on the Canvas.
ViewRoot calls surface.unlockCanvasAndPost(canvas) to schedule
surfaceFlinger::composeSurfaces()
SurfaceFlinger::composeSurfaces() does actual display on panel.
SurfaceFlinger handles transfer of drawn data in canvas to surface front
buffer or backbuffer.

Surface Interface: framework/base/include/ui/ISurface.h
Surface management: framework/base/view/surface.java
Android Graphics System
SurfaceFlinger Architecture
Surface-Flinger
Overlay-Obj1 Overlay-Obj2 OpenGL|ES 1.1
FBDev
Framebuffer device
JVM
Jsr239
OpenGL|ES
Java Binding
3D-Java Apps
Surface-composer
client
Android Java Apps
V4L2-vid pipe 1 V4L2-vid pipe 2
Display Subsystem
video pipe 1 video pipe 2 Graphics pipe
LCD
Android
Surface Output
Video MIO
(TI Adaptation)
PVMF Video
Decoder OMX
TI OMX
Video Decoder
PV Player
Driver
Android Java Apps
JNI Media Class
TI DSP
Codecs
GFX Driver
SGX
Camera HAL
Camera Service
V4L2-camera
driver
TI s/w
PV s/w Google s/w
Open source
TI H/w
Alpha color key
Android
Audio Framework
Android Multimedia
AudioFlinger Architecture
Manages all audio output devices
Processes multiple audio streams into PCM audio out paths
Handles audio routing to various outputs
Available in: libs/external/audioFlinger
HAL available in: hardware/alsa_sound/AudioHardwareALSA.h
APP
APP
Game
Audio
Media
Player
Tone
Audio
Audio
Flinger
Bluetooth
Speaker
Earpeace
Android Multimedia
Audio Flinger Interaction Layer
Android Multimedia
Audio Architecture overview
UI
AudioTrack.java
AudioFlinger.cpp
Audio Policy Service
Audio Policy
ALSA Audio HAL
OMAP3 ALSA lib
ALSA lib
Kernel
Java code
Start Stream
(openTrack)
Streams
Most of changes
applied for clair by FMA
Minor changes
applied for clair by FMA
./ti/omap3/modules/alsa
./hardware/alsa_sound
AudioPolicyManagerALSA.cpp

./frameworks/base/media/java/android/media/AudioTrack.java
IN
OUT
Android Multimedia
Audio Framework Components
Stream: digital bits that represent a sound
Stream VOICE_CALL (used for Voice record)
Stream SYSTEM (for system sounds)
Stream RING
Stream MUSIC
Stream ALARM
Stream NOTIFICATION
Audio Flinger: mixer (mix sound, adjust soft volumes, ): take several streams, mix
them + apply volume on each stream. These streams are entering (INs stream) or
outputing (OUTs stream).
OUTs & INs are physical HW channels
These OUTs & INs are attached to devices.
Device= headset, handset,
Streams are grouped into strategy
Strategy:
Phone: VOICE_CALL, BT_SCO
Sonification: RING, NOTIFICATION, ALARM
ENFORCE_ENABLED: unmutable strings (exple: in some country it is mandatory to produce a
sound when recording with a camera )

Android Multimedia
Audio HAL
HAL definition:
Interface: AudioHardwareInterface.h
Stub version of the HAL is located into the framework directory:
framework/base/libs/audioflinger/AudioHardwareStub.h


Platform implementation is located into hardware directory:
TI HAL solution (depreciated):
hardware/omap3/libaudio/audioHardwareOmap.cpp

ALSA HAL solution (in used):
hardware/alsa_sound/AndroidHardwareAlsa.cpp
Android Multimedia
Android Multimedia
Components Diagram
Media Framework
OpenCore
libopencorehw
TI DSP/OMX
DSP Bridge
OpenMax IL
Camera
Service
OMAP Video Overlay
OMAP Camer
Interface
OMAP Camera
Overlay
ALSA
OMAP Audio
libaudio
Media Server
PV/OMX
ARM Codec
DSP Accelerated
OpenMax codec
libOverlay
OMAP Video Overlay
ARM <-> DSP
TI DSP Firmware
Executed on DSP Core
P
a
c
k
e
V
i
d
e
o
T
I

O
p
e
n
M
a
x

I
L

s
o
l
u
t
i
o
n
A
n
d
r
o
i
d
F
r
a
m
e
w
o
r
k
T
I

A
u
d
i
o

H
A
L
T
I

D
i
s
p
l
a
y

H
A
L
T
I

C
a
m
e
r
a

H
A
L
StageFright
TIHardware
Renderer
OpenMax IL
S
t
a
g
e
F
r
i
g
h
t
Android Multimedia
Android stack overview
Java UI Application (mediaPlayer)
Libmedia_jni
MediaPlayerInterface
Libmedia.so
JAVA JNI interface
Media library
Libmediaplayerservice.so
Binder
MidiFile
Vorbis
Codec
Libopencore.so
OpenCore PVPlayer
Application framework as part of the
Android framework
Delivered by Google Android package
PV OpenCore is an opensource multimedia framework
handling multimedia content except Vorbis and MIDI
External delivery from PacketVideo.
StageFright as been introduced in the last version of clair
Should replace OpenCore into FroYo
Stage
Fright
Android Architecture
OpenCore external library
Based on PacketVideo OpenCORE platform
OpenCORE is based on the commercial CORE of ParcketVideo
for Android. This framework is Open source.
http://www.pv.com/products/android/index.html
Large documentation into opencore/doc directory
Supports standard video, audio, still-frame formats
Non optimized reference codec available in opencore/codecs_v2
Support for hardware / software codec plug-ins using OMX
Openmax headers available in external_libs_v2
Android stack overview
Hardware Accelerator (IVA2/DSP)
UI Framework
Frameworks\base\media\
PV Open Core
External\opencore\
TIs or 3d party OpenMax
Hardware\ti\omx\
Binder IPC MediaPlayerInterface
OMX Core library wrapper
DSP bridge
Application framework as part of the Android framework
Delivered by Google Android package
(mainly in C++)



PV OpenCore is an opensource multimedia framework handle
multimedia content except Vorbis and MIDI
External delivery from PacketVideo



TI OMX implementation to support the IVA2/C64x DSP.
Part of the hardware specific implementation from TI


Android Multimedia
Android AV Playback layers
U
I

F
r
a
m
w
o
r
k
Hardware Acceleratos (IVA2/DSP, Display)
Java UI Application (mediaPlayer)
PV Player Driver
Java API
JNI Media Class
PV Player Engine
PV Player Nodes/MIO Nodes/OMX
P
V

O
p
e
n
C
o
r
e
T
I

O
M
X
Third-party Video Player develop under JAVA

On Android there is two classes (MediaPlayer and VideoView) provide
support to the video player (VideoView based on MediaPlayer)
framework\media\java\android\media\MediaPlayer.java

Native Java Interface Layer. Allow JAVA APIs access to the native
libraries (C++)
JNI:framework\media\java\android\media\JNI\android_media_MediaPlayer.cpp
Implementation:framework\media\java\android\media\libmedia\MediaPlayer.cpp

Load all necessary libraries for pvplayer and creates the proper MIO
Nodes that will interact with the platform specific HW
external\opencorejava\android\playerdriver.cpp

PVPlayer creates and configure the required nodes involved on the AV
Playback pipeline so they can interact with each other then it will
oversee the operation of the pipeline.

PV node wrap software or hardware codecs and they can sink or
source of data. The MIO nodes contain glue code between the Video
Pipeline and the HW (e.g. methods to send buffers to the Display)
Android Multimedia
Media Player Service

Responsibility:
Provide mediaplayer services to the application framework (MediaPlayer) through the
IPC binder.
Create a proper playerdriver to handle the input stream.
The MediaPlayerService supports 4 players:







Create the connection between the MediaPlayer (framework process) and the new
playerdriver (service process)
Manage the audio ( output to the audio framework /Audio cache)
MediaPlayer C++ class interface with the IMediaPlayerService service through binder IPC
PVPlayer
external\opencore\android\playerdriver.cpp
Compilation flag NO_OPENCORE to remove it
Mediaplayer VorbisPlayer
Framework\base\media\libmediaplayerservice\
Vorbisplayer.cpp
MidiFile
Framework\base\media\libmediaplayerservice\
midifile.cpp
binder
binder
binder
StagefrightPlayer
external\opencore\android\playerdriver.cpp
Compilation flag BUILD_WITH_FULL_STAGEFRIGHT
to include it
binder
Android Multimedia
Player Driver Creation
Process B
Process A
Media Player
ServiceManager Binder Driver
MediaPlayer
Service
Get
service(media.player)
MediaPlayerService
marshal proxy
this (mediaplaye)
Datasource(url)
Call Create(getpid(),this,url)
Transact(CREA
TE_URL,data)
Replay an instance of playerdriver
PVPlayer/vorbisPlayer/MidiFile
Android Multimedia
Player Driver
process/service name: media.player (dumpsys media.player)
Interface: android.hardware.IMediaPlayer
Definition: iMediaPlayer.h
Binder proxy: BpMediaPlayer (iMediaPlayer.cpp)
perform the interface calls through the IPC (application process side)





Binder native: BnMediaPlayer Stub of BnMediaPlayerService
(MediaPlayerService.cpp)
Implement the interface (service process side)
Stub as from BnMediaPlayerService s running into the same process



status_t start() {
Parcel data, reply;
data.writeInterfaceToken(IMediaPlayer::getInterfaceDescriptor());
emote()->transact(START, data, &reply);
return reply.readInt32(); }
onTransact(
reply->writeInt32(start());
Android Multimedia
Media Player Driver
Process B
Process A
Media Player
Binder Driver
PVPlayer
driver
prepareAsync()
Transact(PREPARE_SYNC)
Interface with
OpenCore through
PVPlayerInterface
Start()
Transact(START)
Replay iMediaPlayer getInterfaceDescriptor
Transact(PREPARE_SYNC)
Interface with
OpenCore through
PVPlayerInterface
Android Multimedia
Media Player Service UML
1
2
3
1
2
3
Binder : Retrieve appropriate Player
Binder: Abstract the player Control
OpenCore interface: Abstract PV Engine
Android Multimedia
Android OpenCore Player
OpenCore Common library (libopencorecommon)
OSCL
(Operating System Compatibility Library)
PVPlayer
Player Driver
Video Output
Audio Output
Android Player
Player Engine
PVMI (PacketVideo Multimedia
Interface)
Libopencoreplayer.so
Android Multimedia
OpenCore PVPlayer
Wrapper between Android and OpenCore:
Implement binder android.hardware.IMediaPlayer interface for
UI framework connection
Support a playerdriver object (mPlayerDriver) to interface with
OpenCore (PV Player Engine)

header: frameworks\base\include\media\PVPlayer.h
Implementation: external\openCore\android\playerdriver.cpp



Android Multimedia
OpenCore PlayerDriver
Interface with the PVPlayer Engine:
Support PVPlayerInterface: Asynchronous and Synchronous commands
send to PVPlayer Engine (PlayerCommand)
Implement callback for event handeling called by PVPlayer Engine
(PVCommandStatusObserver,PVInformationalEventObserver,
PVErrorEventObserver OpenCore interfaces)


Header/Implementation: external\openCore\android\playerdriver.cpp
Commands definition/implementation:
external\openCore\android\playerdriver.h
Android Multimedia
OpenCore PVPlayer Engine
Heart of PVPlayer:
receives and processes requests from playerDriver
manages the PVMF components required for multimedia playback and related operations.

Define the PVPlayerInterface to allow PlayerDrivers to interface with
Process PlayerDriver commands initiated by API calls asynchronously.
Notify the PlayerDriver from events coming from PVMF
Maintains a state machine modified based on PVPlayerInterface APIs called and
events from PVMF components below

Header: external\openCore\engines\player\include\pv_player_interface.h
Implementation: external\openCore\engines\player\src\pv_player_interface.cpp

Android Multimedia
PVPlayer Engine Class Diagram
Android Multimedia
PVPlayer Engine State Transition Diagram
Android Multimedia
OSCL (Operating System Compatibility Library)

OSCL abstracting the HLOS dependencies to
manage:
Thread (osclproc directory)
memory (osclmemory directory)
file/socket (osclio directory),
etc

OSCL directory: external/opencore/oscl/
Documentation: external/opencore/doc/oscl_html

Android Multimedia
PVMI(PacketVideo Multimedia Interface)
1) PVMF is the OpenCore multimedia framework:
Define the multimedia architecture upon which the PVPlayer engine is based
Define Nodes (based on OSCL) to manage:
files Nodes Responsible to parse media file (mp3, avi)
Nodes available in: in external/opencore/nodes
Parser available in: external/opencore/fileformats
OMX Nodes to communicate with OMX layers
Encoder: external/opencore/nodes/pvomxaudiodecnode/
Decoder: external/opencore/nodes/pvomxvideodecnode/
PVMF directory: external/opencore/pvmi

2) Provide MIO (Media I/O) interfaces
Files MIO: external/opencore/pvmi/media_io
Android Specific for Surface, Camera, Audio MIO:external/opencore/android/

No specific documentation

Android Multimedia
Media Output to Node and Media I/O
2 methods for PVPlayer Engine handles interfacing to platform
specific output media devices :
1. Encapsulate the media device in a PVMF node which PVPlayer
Engine can use directly.
This method minimizes the code between PVPlayerEngine and the
media device interface.
Interface the media device to PVs Media I/O interface.
PVPlayer Engine can use the PVMF node that interfaces PV Media IO to
output
Android Multimedia
PVMI Component Diagram (Decode path)
PVPlayer Engine
MIO Output PVMF Node
Master Core
Parser Node
Opencore
PVMI

Speaker Display
surfaceflinger audioflinger
AudioHardwareInterface
omx audio/video dec node
OMX Core
MM decoder
TI OMX
media input node
DisplayHardwareInterface
TI SW
Google
Packet Video
Opensource
Hardware
Android Multimedia
PVMI Component Diagram (Record path)
Record Applications
media Input node
Master Core
Muxer Node
opencore
Microphone Camera
surfaceflinger
audioflinger
AudioHardwareInterface
omx audio/video enc node
OMX Core
MM Encoder
TI OMX
media output node
CameraHardwareInterface
Camera HAL
Display
Display
Hardware
Interface
Camera Service
TI SW
Google
Packet Video
Opensource
Hardware
Android Multimedia
OpenMax Definition
Definition:
OpenMAX is a royalty-free, cross-platform API that provides
abstractions for routines especially useful for computer
graphics,video, and audio.
Website: http://www.khronos.org/openmax/
OpenMAX AL (Application Layer)
Audio/Video/Image Playback/Recording
OpenMAX IL (Integration Layer)
Blocks of functionality: sources, sinks, filters, etc.
OpenMAX DL (Development Layer)
Low-level building blocks: DCT, LoopFilter, Quantization, etc..
Android Multimedia
OpenMax IL Overview
OpenMax IL API is a component-based media API
Consists of two main segments:
Core API (unique component):
Dynamic loading/unloading components management
Facilitating component communication (Tunneling)
Component API (several components):
Individual blocks of functionality
Can be source, sink, codecs, filters, splitters, mixers, etc..
Android Multimedia
OpenMax IL into OpenCore
Master Core: OpenCore support several OMX vendors solutions:
method of dynamic loading/linking of multiple OMX cores
OMX core methods called through a thin wrapper layer
(mastercore):external/openCore/codecs_v2/omx/omx_common/src/pv_omxmasterco
re.cpp
Avoid co-existence of multiple OMX cores issues
Maintain a cross-vendors OMX component registry
Interface

PV OpenCore provides an implementation of PV OMX core and PV
OMX components in external/openCore/codecs_v2/omx/

OpenCore framework behaves as OpenMax IL client when it
communicates with Vendors OMX core.

OpenCore PVMF OMX Nodes:
Wrapper between the PVPlayer Engine and the OMX components
Directory:external/openCore/nodes/pvomxencnode
Directory:external/openCore/nodes/pvomxvideodecnode
Directory:external/openCore/nodes/pvomxdecnode
Directory:external/openCore/nodes/pvomxbasedecnode





Media Framework
OpenCore
TI DSP/OMX
OpenMax IL
Ffmpeg/OMX
ARM Optimized Codec
DSP Accelerated OpenMax codec
Android Multimedia
OpenMax IL Master Core
Avoid co-existence of multiple OMX cores issues:
Maintain a cross-vendors OMX component registry
Method of dynamic loading/linking of multiple OMX
cores
Define OMXInterface Interface to be
implemented by OMX Core vendors

Implementation:external/openCore/codecs_v2/omx/omx_c
ommon/src/pv_omxmastercore.cpp

Interface definition:
external/openCore/codecs_v2/omx/omx_common/include/om
x_interface.h

Documentation:
external/openCore/doc/omx_core_integration_guide.pdf





Media Framework
OpenCore
TI DSP/OMX
OpenMax IL
Ffmpeg/OMX
ARM Optimized Codec
DSP Accelerated OpenMax codec
OMX Core
master core
TI core wrapper
master core
component
registry
Component query by role from
Client
OMX Core
PV core wrapper
Other vendor OMX?
OMX.TI.AAC.decode
OMX.TI.XYZ.decode
OMX.TI.AAC.decode
First match?
OMX.PV.aacdecode
opencore player decode
opencore author encode
opencore common
PV master core
OMX master registry
Vendor OMX Core wrappers
Android Multimedia
OpenCore OMX Master Core
Android Multimedia
TI OpenMax IL Solution
TI OpenMax IL is a DSP based solution.
TI OMX Core:
hardware/ti/omx/core_plugin/omx_core_pluging/src/ti_omx-
interface.cpp
TI OMX Components:
hardware/ti/omx/[audio,video,imaging]
Interface with the DSP SocketNode through the Bridge
DSP Components:
DSP baseimage including the algorithm/SocketNodes/bridge/bios
are loaded by android init process (init.rc)
The ARM Bridge is loaded as part of the Linux kernel






Android Multimedia
TI OpenMax IL Solution
ARM
DSP
OMX Core
AAC Decoder
OpenMAX Component
MPEG Video Decoder
OpenMAX Component
AAC Decoder
Socket Node
MPEG Video Dec
Socket Node
LCML
MPU Bridge
DSP Bridge
AAC Decoder
Algorithm
MPEG Video Dec
Algorithm
Master OMX Core
PV OpenCore
PVMF OMX audio/video
decoder nodes
PVMF media
output node
AudioFlinger
Audio Hardware
Interface
TI OMX
SurfaceFlinger
Video Hardware
Interface
USN
TI MM Component
External Component
PVMF Parser
node
PVMF media
input node
Android Multimedia
TI OpenMax IL Solution
Algorithm
XDM/XDAIS compliant. Runs on DSP/BIOS.
Socket Node
Algorithm-specific preparation around the algos process() call
TCONF file specifies memory requirement for various codec profiles
USN
Generic data & control processing, including DSP cache coherency operations
Bridge
Loads DSP baseimage and dlls. Manage DSP nodes. DSP Power Management.
Map/unmap buffers. Implements ARM cache coherency APIs.
LCML
Performs common data & control processing for OMX components.
Groups together several Bridge API calls into one. Serves as Bridge abstraction.
OMX
Implements standard OpenMAX interfaces for integration with HLOS MM Framework
Android/PV Master OMX framework is in C++, whereas TIs code is in C
Android Multimedia
Create of AV playback pipeline
Android Multimedia
Rendering through Overlay Interface
Android Multimedia
Overlay Interface

1. Implement the stub funtions in hardware\libhardware\modules\overlay
\overlay.cpp, this will generate a share library liboverlay.trout.so.
2. In SurfaceFlinger, it will create a DisplayHardware instance, and
call DisplayHardware::init() which will create an overlay engine from
liboverlay.so.

3. When we want a OverlayRef from ISurface, we call
ISurface::createOverlay() which will take use of the overlay engine
create in step 2.
4. Create a Overlay from OverlayRef as it in frameworks\base\libs
\surfaceflinger\tests\overlays\overlays.cpp.

TI implementation to support overlay for the openCore video is available
in: Hardwared/ti/omap3/libopencorehw/
http://git.omapzoom.org/?p=platform/hardware/ti/omap3.git;a=tree;f=libopencorehw;h=4
b843e240897931c150f1a0cb25b1785c9e8dab2;hb=HEAD
Android Multimedia
TI Audio Decode Architecture
LCML
TI OMX Core
PV OMX Core
TI OMX
Decoder
TI OMX
AAC Dec/Enc
TI OMX
AMR Dec/Enc
TI SN
Decoder
TI SN
AAC Decoder/
Encoder
TI SN
AMR Decoder/
Encoder
DSP Bridge
PV OMX Node PV OMX Node PV OMX Node
PV
Parser Node
PV AAC
Parser Node
PV AMR
Parser Node
PVPlayer / Recorder Engine
PVPlayer/ PVRecorder Driver
PV Audio
Input/ output
MIO
PV Input/
Output
Node (MIO)
Inpu/Output File
Audio Flinger
ALSA driver
Audio Driver
DSP
ARM
Kernel
User
M
c
B
S
P

D
R
V
I
2
C

D
R
V
I
2
C
I
2
S
P
C
M
T
r
i
t
o
n
2
Earphone
Stereo/ mono
Headphone
2 x
Speakers
Stereo/ mono
Microphones
JNI Media Class / PVPlayer
MediaPlayer Class
Audio HAL
Android Music Record UI Android Music Player UI
MediaRecorder Class
TI s/w
PV s/w Google s/w
Open source
TI H/w
OMX PCM Dec
DASF
CDN
Not- Supported
AudioFlinger provides Audio Routing & Mixing Features.
Android Multimedia
TI Video Decode Architecture
LCML
TI OMX Core
PV OMX Core
TI OMX
MPEG4 Dec
TI OMX
H264 Dec
TI Socket
Node
MPEG4 Dec
TI Socket
Node
H264 Dec
DSP Bridge
PV OMX Node PV OMX Node
PV MPEG4
Parser Node
PVPlayer/PVRecorder Engine
PVPlayer/PVRecorder Driver
PV Video
output MIO
PV Input/
Output
Node (MIO)
Inpu/Output File
Surfaceflinger
V4L2 display
user lib
V4L2 display
DSP
ARM
Kernel
User
Video overlay
obj
Android Video Player UI
TI s/w
PV s/w Google s/w
Open source
TI H/w
DSS
vid1
vid1
gfx
LCD
PV H264
Parser Node
Android Multimedia
AV Sync
Android Multimedia
720 Split Codec Architecture
TI Proprietary Information Strictly Private
Android Multimedia
720 Split Codec Architecture
Choice of Video Codec
Config parser helps determine the video codec based on the video resolution
Load Ittiam (split) codec if resolution is 720p, TI (DSP-only) codec if
resolution is D1 or lower.
Video codec OMX component registers its DSP MHz requirement with the
OMX Resource Manager
Choice of Audio Codec
OMX Core tries loading TIs (DSP) audio codec first. If this fails, then it loads
Ittiams (ARM) audio codec.
Audio codec OMX component registers its DSP MHz requirement with the
OMX Resource Manager
If 720p video codec was loaded earlier, then DSP MHz are not available for
TI audio codec. RM returns error, failing the loading of TIs audio codec.
Requires Video Codec to be loaded before Audio Codec
PacketVideo is evaluating this change to their Framework.