For Verint Internal Use Only

CONFIDENTIAL COMPANY INFORMATION

FOR VERINT INTERNAL USE ONLY
This document contains proprietary and confidential information
Verint Systems Inc. and may not be distributed outside of Verint.

KB Article –Real-Time Status & Monitoring Flows.

1. Real-Time Status & Monitoring Flows (Troubleshooting)

Author: Kondal Gundla

Version: 1.01
Date: October 31st, 2011
Contact: kondal.gundla@verint.com

2. Problem Summary
This document describes the components interaction flows in displaying the Real-
Time status and in Real-Time Monitoring. The information from this doc can be used
as guidance when troubleshooting issues in Real-Time status display and Monitoring.
This document shows the sequence of messages exchanged between components
and important traces to look for in different component log files.

Refer the “Real-Time Monitoring Data Flow” the section in

“I360_Technical_Overview.pdf” for high level Real-Time Flows.
Refer
“KB_Troubleshooting_ScreenRecording_Issues_In_FreeSeating_Environme
nts.docx” for troubleshooting free seating environment issues.

3. MAS and RIS initializationHigh Level Block

Diagram
Here is the high level block diagram showing the components involved in real time
monitoring.
- RTP Web Service and MAS components are part of the Interaction Application
role (app server). In a App server cluster environment, RTP Web Service and
MAS run on all App servers. Portal Browser will point to one app server (directly
or re-directed from Load Balancer) and interact with these components. RTP Web
Service on a particular App server will always use MAS on that same server.
- Recorder Integration Service is enabled when its role is enabled.
- IP/TDM Recorder services are enabled when their services are enabled.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following is the MAS and RIS (Recorder Integration Service) initialization sequence
diagram.
Connect
Request Screen
Screen data

Agent Desktop

Interaction Application Role

Compare Picture/Response
Portal/Player RTP Web Service
Player Register/Unregister

Compare Picture/Response

Recorder Integration Service Role

Agents update
Audio as Streaming RTP RIS MAS
Start/Stop Stream

Monitor on/off

IP/TDM Recorder

IP/TDM Recorder Role

4. MAS and RIS initialization

Following is the MAS and RIS (Recorder Integration Service) initialization sequence
diagram.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

MAS RIS

GetMonitoredSwitches

GetMonitoredSwitchesResponse

GetMonitoredDevices

AgentInfoCallback

GetRealTimeData MAS registers Agent with RIS for status

updates if there is one supervisor
dashboard showing the particular Agent
GetRealTimeDataResponse is opened.
RegisterAgent

CallInfoCallback CallInfoCallbacks only happen if the

agent is on a call when registered. The
CallInfoCallbacks are sent at the
beginning and end of each session.

Figure 1: MAS and RIS Initialization Sequence Diagram

As shown in the Figure 1, when MAS is connected to RIS, MAS and RIS exchange
following information.
 configured switches (data sources)

 Monitored Devices (extensions).

 Get the current status for Real-time

Initial information is exchanged when the MAS is connected to RIS. After that, MAS will
register the Agent(s) with RIS for status updates if the Agent(s) is/are listed on the
supervisor dashboard, where the real-time status is displayed for Agents, in Portal. Once
the Agent is registered, RIS will respond with CallInfoCallBack message whenever there is
a status change for the registered Agent. If the Supervisor dashboard is opened while
the call is in progress for the agent then the duration field will show the total time since
the call started.

Following traces from RIS logs show the above described messages exchange.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Here are the RIS state enumeration values. These enumerations are used in
CallInfoCallback message for the State field to indicate the current state of the Agent.
See above CallInfoCallback message for details.
Unknown(-1)
      LoggedIn(1)
      LoggedOut(0)
      Idle(2)
      Reserved(3)
      Busy(4)
      Ready(5)
      Talking(6)
      Interacting(7)
      Hold(8)
      Wrapup(9)

Following traces are from the MAS logs showing it is connecting to RIS.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

MAS sending the GetMonitoredSwitches request to RIS and processing the

MonitoredSwitches response from RIS.

Similarly other messages (GetMonitoredDevices, AgentInfoCallback, GetRealTimeData

etc.) can be noticed in MAS log file.

Note: Search on ISConnection in MAS logs to see all the messages exchanged with RIS.

5. Supervisor Dashboard Display

Following is the example Supervisor Dashboard Screen shot showing the Agents
status.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following is the sequence diagram showing the messages exchanged when the
supervisor Dashboard is opened. There may be some other messages exchanged but
here only few important messages are showed.

Browser WebApps DAS MAS

GridPlayback/GridPlaybackIndex.aspx

response

Grid/HeaderPage.aspx

response
Grid/GridPage.aspx
GetRealTimeAgentInfo
GetAgent

Returns the agent Info

GetRealTimeAgentInfoResult
response

Grid/AgentSelectorPage.aspx

Figure 2: Supervisor Dashboard display Sequence Diagram

Figure 2: Supervisor Dashboard displayFigure 2 shows the high level messages

exchanged between Brower, WebApps, DAS and MAS components in displaying the
supervisor dashboard with Agents current status.

Use the Fiddler or some HTTP proxy tool on the Browser PC, where the portal is opened
to see all the http requests/responses from/to browser.

Following URL can be used in the portal to open the client console window for showing
the Real-Time notifications. This mechanism can be used if no tools like Fiddler are
available to use.
 Append the “/Ultra/default.aspx?rtLog=1” string after the http://ServerName. For
Example: http://sevm-l1con1//Ultra/default.aspx?rtLog=1
 Then go to supervisor dashboard and notice that new window opens up showing
the Real-Time messages.

Here is the screen image showing the console window showing the Real-Time messages.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following is an example screen image showing the browser http messages in Fiddler. For
this KBA, Fiddle tool is used to show the details of http messages exchanged.

Following traces from WebApps log showing it is trying the get the real-time info.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following traces from DataAccessServices log showing it is trying the get the real-time
info from MAS.

Following traces from MAS log showing it is processing the GetAgent message.

6. Agent Status Update in Supervisor Dashboard

Following is the sequence diagram showing the messages exchanged between
components in communicating the real-time status.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

RTPWebService MAS RIS

Browser

ComparePicture(Source=Web)

ComparePicture(Source=Web)
RegisterAgent

CallInfoCallBack messages CallInfoCallbacks only happen if

the agent is on a call when
registered. The CallInfoCallbacks
ConstructStatefulPictre are sent at the beginning and
end of each session.
FireReceivePicture
ComparePicture result

Figure 3: Real-Time Status Update Sequence Diagram

As shown in the Figure 3, following is the flow.

 MAS registers for the AgentEvent’s. See Figure 1.

 Once MAS registers, RIS notifies MAS with CallinfoCallBack message.

 MAS maintains the real-time status

 Browser creates a ComparePicture (it is a representation of the current known

state of real time) request to RTPWebService. When the RTPWebService returns
with result or timeout then Browser creates a new request to RTPWebService.
Browser always keeps one outstanding ComparePicture request with
RTPWebService to get the status update immediately.
 PictureID in the request is a unique id identifying the particular real-time monitor
session.
 RTPWebService gets the real-time status info from MAS and forwards it to
Browser.

Following are the Fiddler HTTP traces on playback PC showing the HTTP requests and
responses in getting the real time status.

As the Fiddler traces shown, agent (Ran Davidovitch) is on the call.

CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following are the traces from RTPWebService showing the receipt of the
ComparePicture request. Notice the PictureID and Picture Source parameters.
ComparePicture request from different browsers can be identified by the PictureID
parameter. New GUID is generated for PictureID for each Monitor Session.

Following are the traces from RTPWebService showing the response for the
ComparePicture request.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

MAS traces showing the receipt of the ComparePicture request. Notice the ID, which
is the picture ID of the ComparePicture request.

MAS traces shows the CallInfoCallcack message from RIS with the real-time status.

Following traces show MAS sending the ComparePicture response.

7. Live Monitoring
Note that Audio and Screen live monitor works in different way. Following sections
show the sequence diagrams showing the components involved and messages
exchanged in each type of live monitor.

Note that Live Monitor can be initiated in following ways.

 Select Agent, Viewing Detailed Status: View the detailed status information of
all agents in a specific group, and then select the specific agent you want to
monitor from the list (Quality Monitoring>Supervisor Dashboard)
 Type Extension Number: Enter the agent’s extension number
(Contacts>Monitor)
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

 Select Agent Name: Select the group and the agent you want to monitor within
that group (Contacts>Monitor)

Note that all of the above three methods of Monitoring follow almost the same
sequence of messages exchanges.

7.1 Audio Live Monitor

Following sequence diagram shows the messages exchange in Audio Live Monitoring.
IP/TDM
Browser RTPWebService MAS RIS Recorder
Start Live Monitor
PlayerRegisterFullExtension
ComparePicture(Source=Player)
StartStream MONITOR ON
MONITOR ON
message is sent to
StartStreamResponse Recorder only if it is
ComparePictureResponse(Source=Player) recording the extension

PlayerRegisterFullExtensionResponse
Recorder continues to
send RTP Stream until
Start RTP Streaming to playback client
Keep Alive messages Monitor is stopped
exchanged While live
monitor is in progress
PlayerKeepAlive
KeepAlive
HeartbeatStream

Stop Live Monitor

PlayerUnRegister
AbortPicture StopStream MONITOR OFF

PlayerUnRegisterResponse Stop RTP

Streaming

Figure 4: Audio Live Monitor Sequence Diagram

As shown in
IP/TDM
Browser RTPWebService MAS RIS Recorder
Start Live Monitor
PlayerRegisterFullExtension
ComparePicture(Source=Player)
StartStream MONITOR ON
MONITOR ON
message is sent to
StartStreamResponse Recorder only if it is
ComparePictureResponse(Source=Player) recording the extension

PlayerRegisterFullExtensionResponse
Recorder continues to
send RTP Stream until
Start RTP Streaming to playback client
Keep Alive messages Monitor is stopped
exchanged While live
monitor is in progress
PlayerKeepAlive
KeepAlive
HeartbeatStream

Stop Live Monitor

PlayerUnRegister
AbortPicture StopStream MONITOR OFF

PlayerUnRegisterResponse Stop RTP

Streaming

Figure 4, following are the high level steps Audio Live Monitor.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

 Browser/PlaybackClient will initiate the audio live monitor by sending the

PlyerRegisterFullExtension message to RTPWebService. Playback client IP
Address and ports information in the message. These parameters are passed
all the way to IP/TDM Recorder to stream the RTP Data.
 RTPWebService sends the ComparePicture message to MAS with player as
the Source.
 MAS sends StartStream message to RIS to start streaming the audio.
 RIS sends MONITOR ON message to IP/TDM Recorder to start streaming the
RTP packets.
 While the live monitor is in progress, components exchange Keep Alive
message to make sure that Live Monitor is still active.
 To end the live monitor, Browser/PlaybackClient will unregister the player
with RTPWebService to tear down the live monitoring.
 RTPWebService sends AbortPicture message to MAS to close the Live
Monitor.
 MAS sends StopStream to RIS.
 RIS Sends MONITOR OFF message to IP/TDM Recorder to stop the audio RTP
packets streaming.

Following screen shot shows that Browser sent the PlayerRegisterFullExtension to

RTPWebService. Fiddler tool is used to trace the http messages. Note the parameters
(ip address and port numbers) in the message.

Traces from RTPWebService processing the PlayerRegisterFullExtension message.

CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

MAS traces showing the ComparePicture message. Note that Source is Player.

MAS sending the StartStream message to RIS.

Following traces show RIS processing the STartStream message and sending the
MONITOR ON request to IP Recorder.

Following traces show IP Recorder receiving the MONITOR ON message and start
sending the RTP stream.

Browser sending the PlayerKeepAlive message to RTPWebService.

CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

PlayerKeepAlive message processing in RTPWebService.

Traces in MAS related to KeepAlive. Notice that it sent the HeartbeatStream

message to RIS.

RIS traces related to HeartbeatStream message.

Following Fiddler traces show browser calling PlayerUnRegister method on the

RTPWebService to stop the live monitor.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following are the traces from RTPWebService showing the processing of

PlayerUnRegister message. Notice that it sends AbortPicture message to MAS.

Following traces from MAS shows the processing of AbortPicture message and
sending the StopStream message to RIS.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following traces from RIS shows the processing of StopStream message and sending
the MONITOR OFF message to IP Recorder.

Following traces from IP Recorder showing processing of MONITOR OFF message.

7.2 Screen Live Monitor

Note that PlaybackClient gets the Agent PC Name (or IP Address) and Port number
information for Screen Live Monitor as follows:
 RIS (as part of CallInfoCallback message) sends it to MAS
 MAS (as part of GetAgent response message) gives to RTP WebService.
 RTP WebService (as part of ComparePicture response) gives it to
PlaybackClient.

Here is the actual Live Monitor sequence.

CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Browser/PlaybackClient ScreenCaptureModule

Start Live Monitor

Connect

Get FULLSCREEN
Screen Capture Module
automatically sends the updates
FULLSCREEN whenever there is a screen change

Updates

Stop Live Monitor

Disconnect

Figure 5: Screen Live Monitor Sequence Diagram

As shown in

Browser/PlaybackClient ScreenCaptureModule

Start Live Monitor

Connect

Get FULLSCREEN
Screen Capture Module
automatically sends the updates
FULLSCREEN whenever there is a screen change

Updates

Stop Live Monitor

Disconnect

Figure 5, Screen live monitor works differently compared to Audio Live Monitoring.
Here are the steps in Screen Live Monitor.
 Playback Client directly connects to Screen Capture Module wcapw32.exe
program on agent PC.
 Requests the FULLSCREEN packet at the beginning and every 30 seconds
thereafter.
 Screen Capture Module sends the response to FULLSCREEN request and
sends the updates automatically whenever there is a screen change.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

 When the live monitor is stopped, Playback Client closes the socket
connection to Screen Capture Module to end live monitor.

Following traces from the playback client component show that it is initializing the
components for live monitor. Notice the extension and ports information in audio live
monitor component initialization and wrfb url in screen live monitor component
initialization. Notice the agent PC ip address and port number to connect in wrfb url.

Following trace in playback component shows the loading of the filter to handle the
screen live monitor. Notice the wrfb url with Agent PC ip address and port number.

Following traces in playback client component show that it is connected to Screen

Capture Module and requested FULLSCREEN packet.

Following traces in playback client component show that it received the FULLSCREEN
packet.

Following traces in playback client component show that it received the UPDATE
packets.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Following traces in playback client component show the disconnect processing. It

closes the socket connection to Screen Capture Module on disconnect.

Following traces from wcapw32.exe program from Agent PC shows that client is
connected and requested the FULLSCREEN packet. Notice that it sent the
FULLSCREEN packet in response.

8. PrintDebugInfo RTP WebService method

In version V11 SP1, RTP WebService has been enhanced with “PrintDebugInfo”
method to print the internal state information in MAS log file. Open the URL to RTP
WebService and invoke the PrintDebugInfo method.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

In the RTP WebService log file, following trace is logged when the PrintDebugInfo
method is invoked.

As a result PrintDebugInfo method call, MAS logs the internal state information as
follows.
Beginning of the PrintDebugInfo method:

Printing the internal state info:

CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

End of the PrintDebugInfo method.

9. More info and reference

 Fiddler tool can be downloaded from http://www.fiddler2.com/fiddler2
 Refer the “Real-Time Monitoring Data Flow” the section in
“I360_Technical_Overview.pdf” for high level Real-Time Flows.
 Refer
“KB_Troubleshooting_ScreenRecording_Issues_In_FreeSeating_Environments.do
cx” for troubleshooting free seating environment issues.

10. Product Keywords

Real Time Status Display, Real Time Monitoring Flows, Live Monitor Flows, MAS,
Dashboard, RTM and Agent status.
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

11. MAS internals

In version 11.0 real time monitoring was not supported for several configurations
such as agent’s association to LAN DS and dialers.
To support those configurations in version 11.1, MAS internal repository and some of
the MAS-RIS interfaces were changed. Starting from MAS patch KB106507 and RIS
patch KB106506, those changes are merged to version 11.0 as well.
The following figure is a schematic description of entities in MAS’ repository and the
relations between them

Client – represents a WEB client or playback client, and will include several agents for
dashboard or single agent for playback page.
Monitored Agent – represents a single agent from a specific client, and contains links to
one or more employee IDs – an agent’s association in user manager.
Employee ID – can be anything that the agent is associated to: in fixed seating: extension,
workstation, in free seating – pbx id, user name, dialer employee id
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

Agent Base – represents an extension and holds all the information required for real time
monitoring, CTI data etc. One or more employee id can be logged on to a specific AgentBase
– these connections are controlled by AgentInfoCallback from RIS.
In the following log example we see 2 agent events from RIS that represents 2 employee
IDs logged on to the same extension:

Employee id 1: Pbx id 8012 from phone DS 445000001 is logged on to extension 8012

Employee id 2: Workstation lmdesktop1 from LAN DS 445000002 is logged on to the same
extension
Note that ExtensionDS field was added to the agentInfoCallback event to differ from
employee id’s DS (SwitchId field).
At this version we can use PrintDebugInfo from MasTroubleshooting tool to printing MAS
internals. The tool can be found under %IMPACT360SOFTWAREDIR%\MAs\Bin:
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

And this is how MAS’ repository looks after those login events:

There is one extension (AgentBase) – 8012 from DS 44500001, you can see its current
state (Login), number of connected employee IDs (2), screen IP and port etc.
There are 2 employee IDs, both logged on to 8012, and there are no clients – at this time,
no WEB session was opened.
Once a dashboard is opened in portal for an agent that is associated with pbx id 8012, a
new client with one monitored agent will be created, and this monitored agent will point to
employee id 445000001:8012. This employee ID is already logged on to AgentBase
445000001:8012 (after AgentInfoCallback), and this AgentBase will be sent in the picture
back to portal, after sending registration request to RIS:
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY

If a call is already started, or whenever a call will be started, MAS will get a call info event
from RIS for this extension and will update the relevant AgentBase in its repository. At this
time it will look for all of the clients that are connected to this AgentBase and to its logged
on employee IDs and will update them with the new state and data:

Print Debug Info at this point will give the following log:
CONFIDENTIAL COMPANY INFORMATION
FOR VERINT INTERNAL USE ONLY