ConferenceAPI Workflow

Object Model
Application Conference API
Conference
Venue Participant VideoCapabilitySender VideoCapabilityViewer

Venue Service

DirectShow API
Managed DShow FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager
Network Source Decompress Video Render

Rtp API (High Level)

RtpSender RtpListener
RtpSession RtpStream

Start Bar UI Display CapabilitySenders

In Menus Application Conference API
Conference

Display Venues

Venue Service

Venue Participant VideoCapabilitySender VideoCapabilityViewer

DirectShow API
Managed DShow FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager
Network Source Decompress Video Render

Rtp API (High Level)

RtpSender RtpListener
RtpSession RtpStream

Start BarUI
FMain_Load() Attach local objects and event handlers to the Conference object Conference.ctor Conference.Install() Check to see if ConferenceAPI and its depdendencies are properly installed Perform installation if not This allows ConferenceAPI to be used in XCopy Deployment even though ConferenceAPI has COM dependencies that must be registered and state that must be initialized on the client only once like Performance Counters and Event Logs Load A/V Devices & other CapabilitySenders into the Options menu FMain.DisplayVenues() Conference.Venues.get Conference.LocalParticipant.get RtpSession.ctor Set local RtpSession properties, like CName and Name, for this Participant MSR.LST.ConferenceXP.Identity.Identifier() Gets email address from Messenger or creates one from Windows credentials VenueService.GetVenues( Identity.Identifier ) Makes a Web Service call to the Venue Service set in the app.config file to load a list of venues from a central database foreach (Venue in Conference.Venues) Add Venue to Listbox

Join Venue, RtpListener.ctor

Application Conference API
Conference
Venue Participant VideoCapabilitySender VideoCapabilityViewer

Venue.Join() Venue Service

DirectShow API
Managed DShow FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager
Network Source Decompress Video Render

Rtp API (High Level) Start sending SDES RtcpPackets RtpSender new Participant in Venue announcing
Network

RtpListener
RtpSession RtpStream

Join Venue, RtpListener.ctor

Click on .NET & Grids Venue, Clear Listbox & adjust UI (buttons, menus) FMain.listView_ItemActivate Venue.Join() RtpListener.ctor() Randomly choose a RtpSession SSRC (AKA heartbeatSSRC) rtpNetworkListener == NetworkListener.ctor() Start listening for incoming Rtp Packets rtcpNetworkListener == NetworkListener,ctor() Start listening for incoming Rtcp Packets rtcpNetworkSender == NetworkSender.ctor() Get ready to send outgoing Rtcp Packets telling other clients youre here CheckFirewall() Check the Firewall to ensure the proper ports are open InitializePerformanceCounters() InitializeThreads() Continued on next slide

Join Venue, InitializeThreads

FMain.listView_ItemActivate Venue.Join()

RtpListener.ctor()
Continued from previous slide InitializeThreads() RtpListeningThread() Receives incoming Rtp packets from rtpNetworkListener Creates new RtpStream objects as needed Calls RtpStream.newPacket to forward RtpPackets onto the RtpStreams RingBuffer RtcpListeningThread() Receives incoming Rtcp packets from rtcpNetworkListener SDES (Stream Description, describing Rtp traffic) BYE (Stream Leaving, when Rtp traffic is stopping) Others (SR - Sender Report, RR - Receiver Report, APP) Creates new RtpSession objects as needed from SDES Associates RtpStream objects to RtpSession objects from SDES Remove RtpStream and RtpSession objects if BYE MaintenanceThread() Send Rtcp SDES packets for this RtpSession on rtcpNetworkSender Send Rtcp RR packets for this RtpSession on rtcpNetworkSender Remove RtpStream or RtpSession objects if timed out

Rtp Stream Local

1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2

Rtp Stream Remote

7.1 7.2 7.3 8.1 8.2 8.3 9.1 9.2

Rtcp Session Local

SD RR

Rtcp Session Remote

SD RR

Rtcp Session Local

SD RR

Join Venue, ParticipantAdded

Application Conference API
Conference
Venue Participant VideoCapabilitySender VideoCapabilityViewer On RtpSessionAdded () Conference.ParticpantAdded () Participant Displayed in UI

Venue Service

DirectShow API
Managed DShow FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager RtpSession.ctor() Network Source Decompress Video Render RtpListener.RtpSessionAdded()

SDES RtcpPacket comes in from a RtpListener somewhere on the network RtpListener RtpSender
RtpSession

Rtp API (High Level)

Network

RtpStream

Join Venue, ParticipantAdded

RtpListener.RtcpListeningThread() { Receive Rtcp Packet

If ( RtcpPacket == SDES )
If ( ! RtpSessions.ContainsKey ( SDES.SSRC ) ) { RtpListener.RtpSessionAdded Conference.OnRtpSessionAdded

Rtcp Session Local

SD

{ Participant = new Participant( VenueService.GetParticipant ( SDES.CNAME ) ) Conference.ParticipantAdded ( Participant ) FMain.OnParticipantAdded ( Participant ) Add Participant to Form } }

Send Video
Application Conference API
Conference.AutoSend == true After Venue.Join(), Conference calls DefaultVideoCapabilitySender.Play()
VideoCapabilitySender

Venue Service

Conference
Venue Participant VideoCapabilityViewer

FilterGraphManager.ctor() DirectShow API Render filters Add Capture, Compress, and Network Managed DShow NetworkRenderer.ctor() creates an instance FilterGraphManager FilterGraphManager of RtpSender Video Capture Compress Network Render Network Source Decompress Video Render

Video Capture Filters Capture Thread calls RtpSender.Send ( ) 15-30 times a second
RtpSender Network

Rtp API (High Level)

RtpListener RtpSender.ctor() creates a MaintenanceThread RtpSession That starts sending SDES RtcpPackets announcing RtpStream the RtpStream and the RtpSession its related to

Send Video
if (Conference.AutoPlay == true ), this is done automatically after entering Venue Conference.DefaultVideoDevice.Play VideoCapabilitySender.Play MDShow.Device.Send DShow.FilterGraphManager.ctor() Add Capture, Compress, NetRenderer filters Capture filter brings own thread NetRenderer creates RtpSender which starts its own Maintenance thread DShow.FGM.Play

Capture thread obtains video frame from device and

then calls NetRenderer.DoRenderSample which calls RtpSender.Send

RtpSender.MaintenenceThread Get RtpSession information that was set as a static for the process and is shared between RtpListener and all RtpSenders. This contains the CNAME that links the RtpListeners RtpSession to the RtpSenders RtpStreams.

Send SDES in the background which link the RtpStream created by RtpSender to the RtpListeners RtpSession

VenueCapabilityViewer Added
CapabilityViewer icon is rendered over the Participants icon

Application

Conference API
Conference
Venue Conference.CapabilityViewerAdded () Participant VideoCapabilitySender VideoCapabilityViewer

Venue Service

Conference.RtpStreamToCapabilityViewer ( ) CapabilityViewer is associated to a Participant

DirectShow API
Managed DShow FilterGraphManager RtpListener.RtpStreamAdded ( FilterGraphManager ) Video Capture Compress Network Render Network Source ( Decompress Video Render Conference.OnRtpStreamAdded ) SDES RtcpPacket comes in from a Rtp API RtpSender somewhere over the network RtpSender Network

(High Level)
RtpSession RtpStream

RtpListener creates a new RtpStrea RtpListener And associates it with an RtpSessio

VideoCapabilityViewer Added
RtpListener.RtcpListenerThread Receives SDES from RtpSender RtpListener.RtpStreamAdded Associates RtpStream to RtpSession

Conference.RtpStreamToCapabilityViewer
Conference.CapabilityViewerAdded FMain.RefreshImages

RtpListener.RtpListenerThread Forwards packets to RtpStream Packets dropped because RtpStream.Subscribed == false

Play Video
Application Conference API
Conference.AutoPlay == true Conference calls CapabilityViewer.Play() after CapabilityViewerAdded ()
VideoCapabilitySender

Venue Service

Conference
Venue VideoCapabilityViewer.Play () Participant calls MDShow.Play () which calls FilterGraphManager.ctor () VideoCapabilityViewer then NetworkSource.ctor ()

DirectShow API
Managed DShow FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager
Network Source Decompress Video Render

Rtp API (High Level)

RtpSender NetworkSource.ctor() creates a background thread to Process video, which calls RtpStream.Subscribe () to start RtpStream storing RtpPackets in a RingBuffer for later retrieval

NetworkSources thread calls NetworkSourcesinDShow threa RtpStream.NextFrame a loop and passes the data to the other filters for RtpListener decompressing and rendering
RtpSession RtpStream

Play Video

CapabilityViewer.Play MDShowManager.Play Fgm NetSource rtpStream.Subscribe rtpStream.NextFrame rtpPackets-> NetSource -> Decomp -> Render

Leave Venue
User selects Leave Venue

Application Conference API

Conference

Venue Service

CapbilitySenders & CapabilityViewers are stopped VideoCapabilitySender

Venue Participant VideoCapabilityViewer

DirectShow API
Managed DShow Objects are Dispose () ed FilterGraphManager
Video Capture Compress Network Render

FilterGraphManager
Decompress Video Render

Network Source

BYE RtcpPackets are sent for all Rtp API (High Level) originating RtpStreams & RtpSessions RtpSender Network RtpListener
RtpSession RtpStream

Leave Venue
Rtcp BYE for RtpStream & RtpSession

Extras

RtpSender Details
new RtpSender ( ipEndPoint, name, role, mediaType )
rtpSender.Send( byte[] ) RtpSender Diags FEC

RtpPacket RtcpPacket NetworkSender System.Net.Sockets

RtpListener
new RtpListener ( ipEndPoint) event RtpListener.RtpSessionAdded event RtpListener.RtpStreamAdded byte[] = RtpStream.NextFrame()

Received a video image

RtpStream
RtpFrame RingBuffer

RtpStream
RtpFrame RingBuffer Receive the RtpSession Professors video stream

RtpSession Professor has joined the classroom I join the classroom

RtpPacket RtcpPacket

RtpListener
Diags FEC

NetworkListener

System.Net.Sockets

Multicast RTP
What is RTP?
Peer to peer network transport Handles stream & message data Where latency is more important than guaranteed delivery In use: Windows XP RTC, Java Media Framework, Real Networks

Why write an RTP stack?

Multicast optimized threading & buffering Forward Error Correction instead of buffering (low latency)

How?
Managed code w/ .NET object model DirectShow filters for easy conferencing development

Unicast Optimized
RTP Stream 1 1 2 1 RTP Listener 1 1 RTP Listener 2 3 1 RTP Stream 2 2 1 1 3 1 RTP Stream 3 2 1 3

RTP Listener 3

1
2 2

1
2 2

1
2 2

2 RTP Packets

Multicast Optimized
RTP Stream 1 1 2 Ring Buffer 1 3 1 RTP Stream 2 2 Ring Buffer 2 3 1 RTP Stream 3 2 Ring Buffer 3 3

1 RTP Listener 1 1

RTP Packets