UCCX Troubleshooting

Angelique De Vos
Customer Support Engineer
March 2017
Agenda
 Problem Isolation

 UCCX Engine - CUCM Interaction

 UCCX Database and Reporting

 Live Data

 HA Failover

 System Monitoring Tools – CUIC & RTMT

 Finesse

 Supervisor Call Monitoring

 Post Call Treatment

 Call Recording with Mediasense

 Single Sign-On (SSO)

Problem isolation
Clarify – Ask the right questions
• What is the exact issue?
• Who is facing this issue?
• When did this issue start?
• What else?
Isolate
• Break the problem down
• Exclude as many factors as possible to pin it down to a single suspect/component

• Do not jump to conclusions!

• Identify the evidence needed to confirm the root cause
UCCX – CUCM interaction
 UCCX– CUCM integration overview
PSTN
Router/GW
UCCX
Communications Manager
Cluster
Agent
Stations
SIP (5060)
SIP/H323/MGCP
(5060/1720/2427)
LDAP (389 | 3268)
CTI/JTAPI (2748)
AD/LDAP Recording
AXL (443 | 8443)
CTI (12028)
SCCP | SIP (2000 | 5060)
 Inbound PSTN call flow
6 2 1 PSTN
4 3 6 Router/GW
23 5 5 4 5
Agent
Stations
UCCX
1. Inbound PSTN Call to VGW.
CallManager
2. VoIP leg to CUCM.
Supervisor Cluster
3. CUCM routes call to RP registered by UCCX.
Stations
4. UCCX responds with redirect request to available CTI Port.
5. UCCX triggers application, instantiates and executes script. Call is
Agent ICD Extension: 6000 answered upon Accept step executing.
CTI Route Point 8222
6. UCCX provides scripted queuing treatment or identifies an available
CTI Ports: 10001-9988-9990 resource and consult transfers the call to the agents device.
Check Engine status – Partial service?
 Sample call analysis
//Call is offered to CTI RP
2872713: Mar 28 09:15:31.264 IST %MIVR-SS_TEL-7-UNK:Route Connection=[8222::1/(P1-abhi_1)
GCID=(1,7057)->ACTIVE]->OFFERED, reason=1, Event= (P1-abhi_1) 7057/1 CallCtlConnOfferedEv 8222::1
[#827] Cause:100 CallCtlCause:100 CiscoCause:0 FeatReason:12, cause=100, metacode=129, isMaster=true
//CTI Port selected
2872720: Mar 28 09:15:31.266 IST %MIVR-SS_TEL-7-UNK:Route Connection: [8222::1/(P1-abhi_1)
GCID=(1,7057)->ACTIVE]->OFFERED, CTI Port selected: TP[id=1,implId=9989,state=IN_USE]
//Call disconnected at the CTI RP since it’s REDIRECTED
2872734: Mar 28 09:15:31.299 IST %MIVR-SS_TEL-7-UNK:RP[num=8222], conn=[8222::1/(P1-abhi_1)
GCID=(1,7057)->ACTIVE]->DISCONNECTED, event=(P1-abhi_1) 7057/1 CallCtlConnDisconnectedEv 8222::1
[#842] Cause:100 CallCtlCause:210 CiscoCause:0 FeatReason:6, cause=CAUSE_NORMAL[100],
meta=META_CALL_REMOVING_PARTY[131]
 Common call failure scenario - #1
Partition and CSS issue: it is important to note that the original calling number should be
able to reach BOTH THE CTI ROUTE POINT AND THE CTI PORTS.
In the MIVR logs, the following error message is seen:

302849: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-ROUTE_FAILED:Route failed: All Call

ids=JTAPICallContact[id=10,implId=1481/1,state=STATE_RECEIVED_IDX,inbound=true,App
name=sample_1,task=null,session=1000000011,seq
num=0,cn=7420,dn=7420,cgn=1009,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null,ocn=7420,route=RP[num=7420],TP=null,List
of Active Connections=[7420::1/(P1-sydney_cti_1) GCID=(1,1481)->ACTIVE]-
>OFFERED,Extension=7410,Exception=com.cisco.jtapi.InvalidPartyExceptionImpl: Attempt to redirect to an unknown
destination,Failure reason= call will be rejected, CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION=0x8ccc0034::Attempt to
redirect to an unknown
destination,Contact.Reject.reason=TRIGGER_FAIL,(SelectRouteTime,ObtainingIdleChannelTime,RedirectTime=0,0,16)

302850: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION:com.cisco.jtapi.InvalidPartyExceptionImpl: Attempt to redirect to

an unknown destination

302851: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION: at

com.cisco.jtapi.ConnectionImpl.redirect(ConnectionImpl.java:1378)

302852: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-EXCEPTION: at

com.cisco.jtapi.ConnectionImpl.redirect(ConnectionImpl.java:1261)
 Common call failure scenario - #2
Codec problem between caller and the CTI port: Use the reference callctl to identify the issue
with the establishment of media due to codec related issue.

In the MIVR logs, the following error is seen:

485624: Sep 25 12:29:14.958 EEST %MIVR-SS_TEL-7-UNK:CallID:763 MediaId:48650/1

Task:21000000978, CallCtlConnFailed, Inbound call, callctl cause:107,
[5519:VZD_CC/(P1-CTIAdmin_1) GCID=(1,48650)->ACTIVE]->FAILED
Solution: Check the codec requirements and if needed, add transcoders so that the call leg can
be established.

Look at UCCX JTAPI cause codes on the web to get a list of all possible failure
codes.
UCCX Database and
reporting
Logical overview of databases
UCCX
Configuration Datastore

Historical Datastore
CCX DB
Repository Datastore

CM Platform Finesse CUIC

Platform
Platform database instance
Database What does it store? How to query it?

ccm cluster and platform run sql select name,nodeid from processnode
info, configuration
cuic_data CUIC application data run sql select name from cuic_data:cuicuser

db_phx_config Finesse config No connect permissions via CLI, this needs TAC
remote account

• Can be found as “A Cisco DB” when you list the services

• To check replication use “utils dbreplication status”
UCCX database instance
• Can be found as “Cisco Unified CCX Database” when you list the services

Database What does it store? How to query it?

db_cra Historical datastore – run uccx sql db_cra select count(*) from resource
historical data where active (COUNT(*))
Configuration
datastore –
configuration data from
CCX admin (CSQ’s,
applications)
db_cra_repository Repository datastore run uccx sql db_cra_repository select * from
– scripts, prompts, DocumentsFiletbl
documents,…
UCCX database instance
• To check replication use CLI command “utils uccx dbreplication status” or the UCCX
serviceability pages. Active/Connected is a good replication status, Dropped/Timeout indicates a
replication issue
DBReplication commands to check
configuration
• utils ntp status
• utils diagnose test
• utils network connectivity <secondary node>
• utils uccx dbreplication dump configfiles
Configuration changes when subscriber is
down
• Both nodes must be available to commit changes to the CDS (Configuration Datastore)
CCX Administration (MADM) Logs:
%MADM-ADM_CFG-7-UNK:ICDServlet :: Exception occurred
The SUBSCRIBER node of the HA is not available
%MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception:
Exception=java.lang.RuntimeException: The SUBSCRIBER node of the HA is not available

• You can temporarily disable CDS on the subscriber to make changes on the publisher
 UCCX DB replication reset
• Replication can be broken if Subscriber is unavailable for too long and send queues buffer is
exceeded
• Typically 3-4 days (*can vary with load) Alert Raised!
DBReplicationStopped
• Alert will be raised

Tools-> Datastore Control Center->

Replication Servers

Issuing a Reset Replication causes the following to

occur:
1. Remove database replication
utils uccx dbreplication teardown
2. Setup database replication
utils uccx dbreplication setup
3. Perform data repair process
utils uccx dbreplication repair all
Force database sync - CLI

Remote DB

CDS
Overwrite Target = Local! CDS

HDS HDS
RDS RDS

Cluster Reboot + Replication Reset

required!
 Reporting – CUIC datasources
CUIC Data Sources:

Publisher CUIC uses

Subscriber’s db_cra

Subscriber’s CUIC
Also uses Subscriber’s
db_cra
 Historical reporting – no DB available
• If no Database is available, historical data records are written to flat file and alert is raised

Primary Server Secondary Server

M Replication S
! won’t occur !
Alert Raised!
DB DB HistoricalDataWrittenToFiles

If both DBs are

down Engine File System File System
writes records
directly to file

• Fix Database issue.

• Then, manually import records

from file system with Tools-> Historical Reporting
-> File Restore menu from CCX Admin
 Database issues – further analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Unified CCX Database

DB
online.uccx.log file for issues
related to database service

DB REPLICATION
uccx_repl_output_util.log file
for issues related to database
replication or at a minimum
gather these logs for TAC
Reporting and database performance
• Historical reports are running slow
• Overall high CPU

Looking into processes, you will find that ‘uccxoninit ‘ is

using the highest percentage of CPU time
Reporting and database performance
• Review the configured CUIC reports and their frequency

• Check the historical tables size

Reporting and database performance
• If tables have millions of rows, review purge config:

Default of 90 months can be reduced

Live Data
Live data design overview
Live data logs
• Engine logs (MIVR) –ICD_RTDM, SS_RMCM, upto Xdebugging5
• SocketIO Service logs – DEBUG mode
 Socket IO debugs
1. The Socket IO Server receives messages from the JMS bus for the Topic.

0000001076: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

%CCBU_Camel (camel-3) thread #3 - JmsConsumer[ResourceIAQStats]-6-
MessageProducer:
%[message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":
{"resourceId":"agent1","resourceName":
"agent1","resourceState":3,.................
..
room_name=ResourceIAQStats*agent1][room_prefix=ResourceIAQStats][sta
tus=RECEIVED][topic=jms:topic:ResourceIAQStats]: Event Detail Trace
Socket IO debugs
2. Socket IO Server processes the messages and maps them to the appropriate rooms.

0000005789: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

%CCBU_pool-19-thread-1-6-MessageDispatcher:
%[client_count=-
1][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":
{"resourceId":"agent1","resourceName":"agent1",".....
..
room_name=ResourceIAQStats*agent1][socket_io_server_type=WS/WSS][status=EN
QUEUED]: Event Detail Trace
Socket IO debugs
3. SocketIO message dispatcher sending the update to clients

0000005791: 10.106.87.133: Jan 26 2016 18:57:27.008 +0530:

%CCBU_pool-19-thread-1-6-MessageDispatcher:
%[client_count=2][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resour
ceId":"agent1","resourceName":"agent1","resourceState":3,
"durationInStateMillis":426719,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDurati
on":1165394,
"longestTalkDuration":2266049,"avgHoldDuration":0,...................
.....
[room_name=ResourceIAQStats*agent1][socket_io_server_type=WSS][status=SENT]: Event Detail
Trace
 Socket IO debugs
4. On the Browser Side logs (F12) we see the following:

Message received on
socket::{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceId":"a
gent1","resourceName":"agent1","resourceState":3,"durationInStateMillis":264720
,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDuration":1165394,"longest
TalkDuration":2266049,"avgHoldDuration":0,"longestHoldDuration":0,"avgHandleDur
ation":0,"avgWorkDuration":0,"totalTalkTime":6992368,"totalHoldTime":0,"maxRead
yTime":10199923,"avgReadyTime":1156099,
HA Failover
High Availability
CCX Component Redundancy
Reporting
Primary CCX Server Secondary CCX Server • CUIC reporting
Engine always queries
M Engine Engine S
• First started the Slave DB for
becomes Master reporting.
Database Database • This is to
• Primary is
preferred Master Reporting Reporting prevent load on
• Other node will the current
be Standby Finesse Finesse Master DB for
large queries.

Databases
• Comprised of Historical, Agent, Repository Finesse
and Config Datastores (HDS, RDS and CDS) • Finesse always talks to the Master
•DB mastership prefers Engine Master (unless Engine.
DB is down on Engine Master) • Finesse Failover follows Engine
• Data is read from / written to DB Master Failover
• HDS/RDS has Publisher/Subscriber and • Finesse service Restart does not
syncs using Informix Enterprise Replication initiate FAILOVER
• CDS uses distributed transactions (both
databases must be operational for updates)
 HA – Failover detection
• Cluster View Daemon (CVD) sends and monitors heartbeats between CCX nodes

HA over LAN HA over WAN

Heartbeat sent every 500 Heartbeat sent every 1
ms second
TCP Heartbeats
Failover after 5 missed Failover after 10 missed
heartbeats Primary CCX Server Secondary CCX Server heartbeats
Failover initiated within 5 Port Failover initiated within 10
5900
seconds CVD CVD seconds
Port
5900

TCP Heartbeats
HA – Lost Connectivity Secondary CVD detects missing keep-alives from Primary CVD
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD does not receive heartbeat from node for a long period:
Primary CCX Server Secondary CCX Server nodeId=1,dt=2049
.....
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD M ? ? S ? M CVD CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10245
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node
crash:
Engine ! Engine state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node
crash:
Databases Databases state=Convergence State,nodeInfo=Node id=1 ip=192.168.12.5

Reporting Reporting Master Election initiated on Secondary CVD

%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
Finesse Finesse 7631: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:JavaService66:
Cisco Unified CCX Engine on node 1 change master from true to false
7632: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:Post Master
Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1

• Primary lost connectivity Secondary CVD elects Secondary as Master

%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
node 2 change master from false to true
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,
• Master Election initiated name=Cisco Unified CCX Engine, node=2
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine

• Master Election completed

• Failover completed
HAoWAN – Island mode
Primary CVD detects missing Heartbeats
Primary CCX Server Secondary CCX Server and assumes secondaryhas failed
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD does not receive heartbeat from node for a long period: nodeId=2,dt=10197
CVD M M S ? M CVD %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node
crash:
state=Convergence State,nodeInfo=Node id=2 ip=192.168.13.5
Engine Engine
!
Secondary CVD detects missing Heartbeats
Databases Databases and assumes Primary has failed
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10242
Reporting Reporting %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node
crash:
state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
Finesse Finesse
Secondary CVD performs Master Election
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
%MCVD-CLUSTER_MGR-7-UNK:JavaService66:

• WAN network failure Cisco Unified CCX Engine on node 1 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master
Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1

• Missing Heartbeats detected Secondary CVD elects Secondary as Master

• Master Election performed
• Failover completed
%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
node 2 change master from false to true
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,
name=Cisco Unified CCX Engine, node=2
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
HAoWAN – Network restored
Primary CVD detects Secondary and keeps Master
%MCVD-CVD-7-UNK:Split after network partition is detected, new
nodeId=2
Primary CCX Server Secondary CCX Server %MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
because primaryEngineComputerName=UC85CCXPRI
CVD M M M ? S CVD %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine conditional-
Secondary CVD detects Primary and drops Master
Keep-LocalMaster
from localServiceCisco Unified
%MCVD-CVD-7-UNK:Split CCX Engine
after network on node
partition 1
is detected, new
Engine Engine
! nodeId=1
%MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
Databases Databases because primaryEngineComputerName=UC85CCXPRI
%MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine
Reporting Reporting Secondary CVDfrom
DropLocalMaster performs Master election and drops
localServiceCisco Unified CCX Engine on node 2,
Master
conditional=true
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
Finesse Finesse
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified
CCX Engine on
• WAN network restored node 2 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event:
MASTER_DROPPED,
• CVDs detect dual Masters name=Cisco Unified CCX Engine, node=2
Secondary CVD completes Master election
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
• Master Election performed CONVERGENCE_COMPLETED, name=Cisco Unified CCX
Engine

• Failover completed
 HA Failover issue – further analysis
From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Unified CCX Cluster View
Daemon
System monitoring tools
CUIC – Utilization monitoring
CUIC provides stock reports which lets admins view license utilization and monitor system
usage:
• License Utilization Hourly report: hourly breakdown about number of inbound and
outbound ports used, as well as agent seats

• Application Performance Analysis report: information about calls presented, calls

handled, abandon rate per hour and also the average call duration.

• Aborted and rejected report: The Aborted and Rejected Call Detail Report
provides information about each call that is aborted or rejected by the system
CUIC – Aborted and rejected report
• Incoming Call Contacts are Classified as Aborted if an exception occurs in the workflow that is
processing a call.
• Incoming Call Contacts are Classified as Rejected when a UCCX Application system resources
reach maximum capacity
• Database Contains a Cause Code for Abort/Reject that Can Alert Admins to Problems
• Reject - TRIGGER_MAX_SESSION
• Reject - REMOTE_TIMEOUT
• Reject - CHANNELS BUSY
• Aborted – Too many transfer failures
• Aborted – Max Steps Executed
Reports give helpful contact
details and exact timestamps!
RTMT – System summary
RTMT – Alert Central
Catch Problems Before
Users or Customers Do!
RTMT – Schedule log collection
Intermittent Issue?
Finesse
Finesse health
• Finesse depends on the following services for its normal
functioning :
• Cisco Unified CCX Engine Service
• Cisco Unified CCX Notification Service
• Cisco Finesse Tomcat (Tomcat Service exclusive to Finesse)

• If any of the above services is not running or in partial

service state (viewed from the CCX Serviceability UI or
using CLI), Finesse will not function properly.
• The service state on GUI shows the state of the Finesse
Tomcat only and is not an indication of the state of
Cisco Finesse. Finesse health can be checked using the
URL:
https://<UCCX IP>:8445/finesse/api/SystemInfo
Finesse failover
Scenario CCX HA Behavior Finesse Server HA Finesse Client Recovery
Behavior Behavior
CCX Engine Failure Failover to HA node Failover to HA node Failover to HA node Finesse follows
Engine mastership
CCX Notification No Failover Finesse goes Out of Sessions closed Finesse unavailable
Service Failure Service until Notification
Service comes online
Finesse Tomcat No Failover Finesse goes Out of Sessions closed Finesse unavailable
Failure Service until Tomcat Service
comes online

Finesse Service OOS No Failover Finesse goes Out of Sessions closed Finesse unavailable
Service until issue fixed
Island Mode Both HA nodes Both Finesse servers Clients connect to Clients reconnect to
become Master In Service either Master node
 Finesse issues – further analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Finesse, Cisco
Unified CCX Engine, Cisco Unified CCX Notification Service

Finesse Logs Notification Service Logs

webservices, client-logs, Needs to be explicitly
realm log file for issues enabled from CLI.
related to Finesse For all login issues, state
change issues on Finesse
Engine Logs
MIVR logs, Axlclient logs,
for login issues
Supervisor Call Monitoring
Monitoring overview
Supervisor phone places
Finesse monitoring call to agent phone Agent Phone
Desktop Supervisor Phone
V
(Browser)
Agent phone connected with caller

UCCX Server

Finesse Server
(Tomcat) SIP/SCCP

SUPERVISE_CALL_REQ with
Supervisory Action field of
SUPERVISOR_MONITOR

CTI
RmCm JTAPI startMonitor method call with agent device info CUCM
Server

UCCX Engine
Monitoring overview
Cisco Finesse Server Receives Silent Monitor request from Finesse client browser
Finesse Web Services log

15:35:34.122 -0500: %CCBU_http-bio-8082-exec-9-6-API_REQUEST:

%[REQUEST_URL=User/agent1/Dialogs][agent_id=agent1][requestId=569bdab5-56a0-4964-b233-
32f515a86cc3][request_method=user.POST][request_parameters= fromAddress:2001
toAddress:2002 requestedAction:SILENT_MONITOR]: Request from client to webservice api
Monitoring overview
Cisco Finesse Server Sends Silent Monitoring Request to UCCX Engine
Finesse Web Services log

15:35:34.123 -0500: %CCBU_pool-12-thread-21-6-

MESSAGE_TO_CTI_SERVER: %[cti_message=invokeID=6576, agentConnectionCallId=-1,
supervisorConnectionCallID=-1, supervisoryAction=1, agentExt=2002, supervisorExt=2001]
[cti_message_name=SuperviseCallReq]: Message going to the backend cti server
Monitoring overview
UCCX Engine Receives Silent Monitor Request from Finesse Server
UCCX Engine (MIVR) Log

15:35:34.126 EST %MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg

Socket:Socket[addr=127.0.0.1,port=35684,localport=12028] invokeID:6576 Msg Type =
SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=6576
AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType
= 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR,
AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null,
AgentDevice = 2002, SupervisorDevice = 2001
Monitoring overview
UCCX Engine Begins Silent Monitoring Request to CUCM via JTAPI
UCCX Engine (MIVR) Log

1221494: Jan 14 15:35:34.129 EST %MIVR-SS_RM-3-Initiating silent monitor request to JTAPI

server...:Undefined mnemonic 'Initiating silent monitor request to JTAPI server...'
Monitoring overview
JTAPI Calls CallStartMonitoring Request Method to Invoke Monitoring on CUCM
JTAPI Log

15:35:34.131 EST %JTAPI-PROTOCOL-7-UNK:(P2-14.10.191.3)

[MIVR_SS_RM_TPCCPROCESSOR-219-12-SupervisorMonitorReqMsgHandler] sending:
com.cisco.cti.protocol.LineCallStartMonitoringRequest { sequenceNumber = 95
lineCallManagerID = 1 lineID = 190 globalCallManagerID = 1 globalCallID
= 1105 callingAddress = 2001 targetAddress = 2002 targetdeviceName
= SEP9CAFCAFFCBD5 targetCallID = 29117566 monitorMode =1
playToneDirection = 3 }
Monitoring overview
JTAPI Shows Monitoring Call Setup Events
JTAPI Log

15:35:34.510 EST %JTAPI-PROTOCOL-7-UNK:(P2-14.10.191.3) received Event:

com.cisco.cti.protocol.CallMonitoringStartedEvent { eventSequence = 543
lineCallManagerID = 1 lineID = 194 callCallManagerID = 1 callLegID
= 29117566 monitorMode = 1 activeToneDirection = 3 }
Monitoring overview
UCCX Engine Sends Confirmation to Finesse Server that Monitoring has Started
UCCX Engine Log

1221618: Jan 14 15:35:34.573 EST %MIVR-ICD_CTI-7-UNK:MsgHandler : Sent : {

SUPERVISOR_CALL_CONF to Socket[addr=127.0.0.1,port=35684,localport=12028] }
Monitoring overview
Finesse Server Receives Monitoring Confirmation Back from UCCX Engine
Finesse WebServices Log

15:35:34.684 -0500: %CCBU_CTIMessageEventExecutor-0

-6-DECODED_MESSAGE_FROM_CTI_SERVER:
%[cti_message=com.cisco.ccbu.finesse.adapter.acmi.message.CTISuperviseCallConf@163c08b
[connectionCallId=16778321,connectionDeviceIdType=0,connectionDeviceId=2002,invokeID=6576
,msgID=125,timeTracker=
{"id":"SuperviseCallConf","CTI_MSG_NOTIFIED":1452803734684,"CTI_MSG_RECEIVED":14528037346
83},msgName=SuperviseCallConf,deploymentType=CCX]][cti_response_time=1]: Decoded Message
to Finesse from backend cti server
Monitoring troubleshooting #1
Problem: Start Monitoring Button Is Not Enabled
• Agent to be monitored needs to be in a Talking state (Not Ready on a call doesn’t work)
• Supervisor’s phone must be on hook and supervisor must be in the Not
Ready state
Monitoring troubleshooting #2
Problem: Monitoring Fails to Start

CCX Engine Receives Request to Silent Monitor agent106

CCX Engine log:

%MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028]

invokeID:3081 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3081
AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0,
SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR, AgentConnectionDeviceID = 2002,
SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice = 2002, SupervisorDevice = 2001
Monitoring troubleshooting
Problem: Monitoring Fails to Start
#2

CCX Engine Runs a Check to see if the agent to be monitored is already being monitored
CCX Engine log:

%MIVR-SS_RM-7-UNK:thisAgent.isSilentMonitored() is true. returning...

Monitoring troubleshooting
Problem: Monitoring Fails to Start
#2

CCX Engine sends error code and text to Finesse

CCX Engine log:

%MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

type=CONTROL_FAILURE_CONF,invokeId=3081,failureCode=CF_RESOURCE_BUSY,errorCode=88056, text=Agent with ID:
agent106 is already in another silent monitoring.
Monitoring troubleshooting
Problem: Monitoring Fails to Start
#3

CCX Engine Receives Request to Silent Monitor agent106

CCX Engine log:

%MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028]

invokeID:3164 Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3164
AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0,
SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR, AgentConnectionDeviceID = 2002,
SupervisorConnectionDeviceID = 1004, AgentID = null, AgentDevice = 2002, SupervisorDevice = 1004
Monitoring troubleshooting
Problem: Monitoring Fails to Start
#3

CCX Engine Invokes Monitoring Request via JTAPI but displays error
CCX Engine log:

%MIVR-SS_RM-3-Initiating silemt monitor request to JTAPI server...:Undefined mnemonic 'Initiating silent monitor request to JTAPI
server...‘
…..
%MIVR-SS_RM-7-UNK: JTAPI error code:88046
Monitoring troubleshooting
Problem: Monitoring Fails to Start
#3

CCX Engine Reports Error Back to Finesse Server

CCX Engine log:

39332: Feb 23 15:05:41.144 EST %MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

type=CONTROL_FAILURE_CONF,invokeId=3175,failureCode=CF_GENERIC_UNSPECIFIED,errorCode=88046, text=Error from
Supervisor Monitor request. Description: ICDJtapiCallControlChannel (monitor): Agent Phone has no BiB Configured. to socket:
Socket[addr=127.0.0.1,port=35064,localport=12028]
Supervisor Barge In
Barge In Overview
• Allows a supervisor to join a call between an agent and a caller
• Supervisor must first be succesfully monitoring agent call
• Supervisor clicks the barge-in button enabled on the monitoring call
• UCCX Engine instructs Agent Phone via JTAPI to place caller on hold and dial the
supervisor phone
• On answer (auto), the supervisor is conferenced into the agent call.
Barge In - Troubleshooting

Supervisor is
monitoring agent call
and clicks Barge In

A delay is observed,
A delayby
followed is observed,
Error
followed by Error

The supervisor later

goes to a Reserved
State
Barge In - Troubleshooting
Supervisor is
monitoring agent call
and clicks Barge In

A delay is observed,
Finesse Server Requests Barge-In to UCCX Engine followed by Error
Finesse Desktop WebServices log and UCCX Engine log:

%CCBU_http-bio-8082-exec-11-6-API_REQUEST: %[REQUEST_URL=User/agent3/Dialogs][agent_id=agent3][requestId=d4bd0140-6be5-
4e75-9e00-7a20e47e2e1d][request_method=user.POST][request_parameters= fromAddress:1004 toAddress:2002
requestedAction:BARGE_CALL associatedDialogUri: /finesse/api/Dialog/16779268]: Request from client to webservice api

%MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3404

Msg Type = SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3404 AgentConnectionCallID =
16779279, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0, SupervisorConnectionDeviceIDType = 0, SupervisoryAction =
SUPERVISOR_BARGE_IN, AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice = 2002,
SupervisorDevice = 2001
 Barge In - Troubleshooting
Supervisor is
monitoring agent call
and clicks Barge In

Finesse Server Requests Barge-In to UCCX Engine A delay is observed,

UCCX Engine log and Finesse WebServices log: followed by Error

%MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

type=CONTROL_FAILURE_CONF,invokeId=3404,failureCode=CF_GENERIC_UNSPECIFIED,errorCode=0, text=Error from Supervisor Barge-In
request. State of the request during failue: WAIT_FOR_SESSION_OFFERED Failed to get notification from Supervisor agent after Consult to
socket: Socket[addr=127.0.0.1,port=35064,localport=12028]

%CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIControlFailureConf [failureCode=0,

peripheralErrorCode=0, text=Error from Supervisor Barge-In request. State of the request during failue: WAIT_FOR_SESSION_OFFERED
Failed to get notification from Supervisor agent after Consult]CTIMessageBean [invokeID=3404, msgID=35,
timeTracker={"id":"ControlFailureConf","CTI_MSG_NOTIFIED":1456263587009,"CTI_MSG_RECEIVED":1456263587009},
msgName=ControlFailureConf, deploymentType=CCX]][cti_response_time=0]: Decoded Message to Finesse from backend cti server
 Barge In - Troubleshooting
A delay is observed,
followed by Error

10 seconds later, Finesse Server Notifies Client of Failure via XMPP (OpenFire) A delay is observed,
Finesse WebServices log: followed by Error

%CCBU_pool-12-thread-5-3-API_ERROR: %[error_code=1][error_message=Generic Barge Call Error]: Error processing REST API request

%CCBU_pool-12-thread-5-6-XMPP_PUBLISH_ASYNCHRONOUS: %[NodeId=/finesse/api/User/agent1/Dialogs][Payload=&lt;Update&gt; &lt;data&gt;

&lt;apiErrors&gt; &lt;apiError&gt; &lt;errorData&gt;1&lt;/errorData&gt; &lt;errorMessage&gt;Generic Barge Call Error&lt;/errorMessage&gt;
&lt;errorType&gt;Generic Error&lt;/errorType&gt; &lt;/apiError&gt; &lt;/apiErrors&gt; &lt;/data&gt; &lt;event&gt;post&lt;/event&gt;
&lt;requestId&gt;ae5bf830-0713-45c6-b085-5810da44ab94&lt;/requestId&gt;
&lt;source&gt;/finesse/api/User/agent1/Dialogs&lt;/source&gt;&lt;/Update&gt;]: Publishing XMPP Message Asynchronously
 Barge In - Troubleshooting
A delay is observed,
followed by Error

5 seconds later, CCX Engine Notifies Finesse Server of New Incoming Call to Supervisor A delay is observed,
Finesse WebServices log: followed by Error

%CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER:
%[cti_message=com.cisco.ccbu.finesse.adapter.acmi.message.CTICallOriginatedEvent@dd1de[peripheralId=1,connectionDeviceIdType=0,callId=167
79284,skillGroupNumber=1,callingDeviceType=76,calledDeviceType=76,localConnectionState=1,eventCause=-
1,connectionDeviceId=2002,callingDeviceId=2002,calledDeviceId=2001,fltReqMaskNumMasks=1,fltReqMaskInstrumentList=[2002],fltReqMaskExtensi
onList=[2002],fltReqMaskCallIDList=[16779284],fltReqMaskFlagsList=[0000000000000000],fltReqCallMaskList=[0804000000000000],fltReqAgentMas
kList=[0000000000000000],invokeID=<null>,msgID=15,timeTracker={"id":"CallOriginatedEvent","CTI_MSG_RECEIVED":1456263592134,"CTI_MSG_
DISPATCH":1456263592135},msgName=CallOriginatedEvent,deploymentType=CCX]][cti_response_time=1]: Decoded Message to Finesse from
backend cti server
 Barge In - Troubleshooting
The supervisor later
goes to a Reserved
State

Finesse Server Notifies Finesse Client that Supervisor is receiving a new call A delay is observed,
Finesse WebServices log: followed by Error

%CCBU_pool-12-thread-12-6-XMPP_PUBLISH_ASYNCHRONOUS: %[NodeId=/finesse/api/User/agent1][Payload=&lt;Update&gt; &lt;data&gt;

&lt;user&gt; &lt;dialogs&gt;/finesse/api/User/agent1/Dialogs&lt;/dialogs&gt; &lt;extension&gt;2001&lt;/extension&gt;
&lt;firstName&gt;Agent&lt;/firstName&gt; &lt;lastName&gt;One&lt;/lastName&gt; &lt;loginId&gt;agent1&lt;/loginId&gt;
&lt;loginName&gt;agent1&lt;/loginName&gt; &lt;pendingState&gt;&lt;/pendingState&gt; &lt;roles&gt; &lt;role&gt;Agent&lt;/role&gt;
&lt;role&gt;Supervisor&lt;/role&gt; &lt;/roles&gt; &lt;settings&gt; &lt;wrapUpOnIncoming&gt;&lt;/wrapUpOnIncoming&gt; &lt;/settings&gt;
&lt;state&gt;RESERVED&lt;/state&gt; &lt;stateChangeTime&gt;2016-02-23T21:39:52.232Z&lt;/stateChangeTime&gt;
&lt;teamId&gt;1&lt;/teamId&gt; &lt;teamName&gt;Default&lt;/teamName&gt; &lt;teams&gt; &lt;Team&gt; &lt;id&gt;1&lt;/id&gt;
&lt;name&gt;Default&lt;/name&gt; &lt;uri&gt;/finesse/api/Team/1&lt;/uri&gt; &lt;/Team&gt; &lt;/teams&gt;
&lt;uri&gt;/finesse/api/User/agent1&lt;/uri&gt; &lt;/user&gt; &lt;/data&gt; &lt;event&gt;PUT&lt;/event&gt; &lt;requestId&gt;&lt;/requestId&gt;
&lt;source&gt;/finesse/api/User/agent1&lt;/source&gt;&lt;/Update&gt;]: Publishing XMPP Message Asynchronously
Post Call Treatment
Post Call Treatment Overview
How it works:

1. Agent clicks End Call via Finesse

2. When CCX Engine Receives this CTI Request to end call, it first runs a check for a call variable
attached to the call called PostCallTreatment
3. If it finds this variable, it then checks to make sure the agent ending the call is the only agent
connected to the caller
4. If a DN is found in this variable, it performs a transfer to that DN rather than ending the call
Post Call Treatment – Script review
• 1. Verify the script has a variable of type int named something like PostCallTreatment with the
correct Value

2. Verify that this variable is mapped to an ECC variable named exactly --PostCallTreatment--
Post Call Treatment - Logs
UCCX Engine Script Debugging Shows Script Variable Set Correctly
UCCX Engine Log (MIVR):

An object of com.cisco.wfframework.obj.WFWorkflowContext { {resourceID, java.lang.String, ""} {CSQ, java.lang.String,

""} {DelayWhileQueued, java.lang.Integer, 30} {WelcomePrompt, com.cisco.prompt.Playable, SP[ICD\ICDWelcome.wav]}
{QueuePrompt, com.cisco.prompt.Playable, SP[ICD\ICDQueue.wav]} {SRS_TempResourceSelectedVar2,
com.cisco.user.User, null} {PostCallTreatment, java.lang.Integer, 1071} }

Most importantly, UCCX Engine Shows the PostCallTreatment variable attached to the call is set correctly
UCCX Engine Log:

%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_DELIVERED_EVENT: Socket:Socket: null monitoredDeviceDN:2082,

connectionCallID: 16779415, lineType: 3, skillGroupNumber: 1, serviceNumber: 0, alertingDeviceType: 73, callingDeviceType:
0, calledDeviceType: 73, lastRedirectDeviceType: 76, localConnectionState: 2, eventCause: 22, connectionDeviceID: 2082,
alertingDeviceID: 2082, callingDeviceID: 1003, calledDeviceID: 2082, lastRedirectDeviceID: 2002, secondaryCallID: 0, ani:
1003, dnis: 1071, userToUserInfo: null, dialedNumber: 1071, callerEnteredDigits: null, callVar1: voice1, callVar2: null, callVar3:
null, callVar4: null, callVar5: null, callVar6: null, callVar7: null, callVar8: null, callVar9: null, callVar10: null, wrapupData: null,
PostCallTreatment: 1071, RemaskField1 : CRACTIRemaskField [reqMaskInstrument=2002, reqMaskExtension=2002,
reqMaskCallID=16779415, reqMaskFlags=[0, 0, 0, 0, 0, 0, 0, 0], reqCallMask=[0, 0, 0, 0, 0, 0, 4, 88], reqAgentMask=[0, 0, 0, 0,
0, 0, 0, 0]] } to outboundQ
Post Call Treatment - Logs
UCCX Engine Receives Agent Hangup Request from Finesse Server
UCCX Engine Log:

%MIVR-SS_RM-7-UNK:Processing msg: CTIClearConnectionReqMsg CTIMgrTPCCReqMsg

Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:6310, peripheralId:1, connectionCallId:16779415,
connectionDeviceIdType: 65535, connectionDeviceId: 2002, agentInstrument: null

Instead of immediately going to JTAPI to disconnect the call on the agent phone, run PostCallSurvey checks:
UCCX Engine Log:

%MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler - isPostCallSurveyEnabled postCallSurveyDN: 1071728797: Feb 24

16:27:53.283 EST %MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler:runHandler connectedAgents.size: 1728798: Feb 24
16:27:53.283 EST %MIVR-SS_RM-7-UNK:ClearConnectionReqMsgHandler - isPostCallSurveyEnabled. Only agent. Transferring
the call to survey
Call Recording with
Mediasense
 Subscribed Application
MediaSense recording – CUCM
1. Caller dials in, CUCM sets up call 4. CUCM Sends 2 Invites to MediaSense
across SIP Trunk
6.Call Control Service sends RTP ports in 11. API Service Sends
SIP 200 OK Reply SESSION_STARTED_EVENT
CUCM Subscribed App

Call Control
3.CUCM Knows that Phone Has a API Service
Recording Profile Attached Service
9. Call Control Service sends metadata to
API Service 10. API Service writes to DB
7.CUCM communicates RTP ports for
forking stream to the phone

Media Service Database

Service
5.Call Control Service notifies
8. Phone establishes RTP with Media Media Service of incoming
Service recording. MediaService
2. RTP established between caller and
phone chooses 2 RTP Ports.
Mediasense recording – Finesse Supervisor Agent
Customer Voice RTP Stream Desktop Desktop
Agent Voice RTP Stream
REST API Calls
JTAPI Finesse
UCCX
CTI Requests (ACMI) Server
ACMI
UCCX Engine
CTI
RmCm
CUCM
Server

SIP
MGCP/H323
SIP/SCCP
BIB Subscription

PSTN/WAN

Customer Agent Phone MediaSense

Voice Gateway
Mediasense Troubleshooting
Is CUCM
Automatic
recording Yes
CLOSED_ERROR working?
status
No recording found

Test call Review

Check
MediaSe Finesse
network
nse Success workflow
Busy
Verify
Verify device and Review
MS
Analyze MS logs line configuration UCCX
service
licensing
status

Verify Check UCCX +

CUCM Finesse logs
config

Analyze CUCM +
MS logs
Is CUCM Automatic Call Recording working?
• Isolate the issue by taking UCCX and Finesse out of the flow, on the CUCM agent line
configuration, enable automatic call recording:

1. Closed_Error status

2. Recordings cannot be found

Mediasense Troubleshooting
Is CUCM
Automatic
recording Yes
CLOSED_ERROR working?
status
No recording found

Test call Review

Check
MediaSe Finesse
network
nse Success workflow
Busy
Verify
Verify device and Review
MS
Analyze MS logs line configuration UCCX
service
licensing
status

Verify Check UCCX +

CUCM Finesse logs
config

Analyze CUCM +
MS logs
CLOSED_ERROR
Cause can be identified from logs or metadata

Error Detail Code in metadata and Closed_Error Reason in Description

logs Search and Play
MEDIA_SERVER_ERROR Record Response Fail Call control service got an error response from the Media
(recording) server for the open or close request.

MEDIA_SERVER_TIMEOUT Record Response Timeout Call control service timed out waiting for response from
the Media (recording) server for the open or close
request.

SIP_SIGNALING_ERROR SIP Signal Error Call control service detected a SIP signaling error, for
example a missing ACK.

SIP_CANCEL_RECEIVED Record Cancelled Recording was canceled by call control service, such as
CANCEL or premature BYE received from CUCM.
CLOSED_ERROR
Cause can be identified from logs or metadata
Error Detail Code in metadata Closed Error Reason in Description
and logs Search and Play

NO_MEDIA_RECEIVED Zero Size Tracks Session was successfully closed, but ALL tracks have zero
size.

ORPHANED Orphaned Session Session was orphaned, e.g., forcibly closed after service
restart.

UNSUPPORTED_CODEC Unsupported Codec Unsupported Codec used to record session.

Check MediaSense logs
Mediasense Troubleshooting
Is CUCM
Automatic
recording Yes
CLOSED_ERROR working?
status
No recording found

Test call Review

Check
MediaSe Finesse
network
nse Success workflow
Busy
Verify
Verify device and Review
MS
Analyze MS logs line configuration UCCX
service
licensing
status

Verify Check UCCX +

CUCM Finesse logs
config

Analyze CUCM +
MS logs
Test call to MediaSense
• Dial the Route Pattern that points to to the MediaSense SIP trunk from an IP
Phone
• A valid recording, with just a single participant (the calling phone) should be
successfully created:
Verify MS service status
• Check if all services are running from Control Center – Feature services
 Verify MS service status
Show Tech in CLI
Use ‘show tech
call_control_service’ from
each node in the CLI

Shows Useful System Info

• System Info
• Status of All Adapters
• Extensive Recording Stats
• Local Storage Info
 Verify CUCM configuration
Route List
MediaSense
Route group

Route pattern Route

SIPgroup
Trunk
MediaSense
SIP Trunk

Recording profile
Mediasense Troubleshooting
Is CUCM
Automatic
recording Yes
CLOSED_ERROR working?
status
No recording found

Test call Review

Check
MediaSe Finesse
network
nse Success workflow
Busy
Verify
Verify device and Review
MS
Analyze MS logs line configuration UCCX
service
licensing
status

Verify Check UCCX +

CUCM Finesse logs
config

Analyze CUCM +
MS logs
Verify device and line configuration
• Ensure that a supported device is being used and that Builtin Bridge is enabled on the device
configuration

• Ensure Line has the correct Recording Profile

Mediasense Troubleshooting
Is CUCM
Automatic
recording Yes
CLOSED_ERROR working?
status
No recording found

Test call Review

Check
MediaSe Finesse
network
nse Success workflow
Busy
Verify
Verify device and Review
MS
Analyze MS logs line configuration UCCX
service
licensing
status

Verify Check UCCX +

CUCM Finesse logs
config

Analyze CUCM +
MS logs
Review Finesse workflow
• Verify if CUCM has selective recording selected

• Verify action configuration on Finesse admin pages

Review Finesse workflow
• Verify workflow configuration
Review UCCX licensing
• Recording ports in MediaSense are not enforced in MS software
• Instead, UCCX enforces simultaneous recordings
Single Sign On
SSO overview
Applications/logs for troubleshooting
Application/log Details
Cisco IdS log Cisco IdS logger will log any
error that happened in Cisco IdS
Fedlet logs Fedlet logs will give more details
about any SAML errors that
happens in Cisco IdS
Cisco IdS API metrics API metrics can be used to look
into and validate any errors that
Cisco IdS APIs may have
returned and number of requests
that are processed by Cisco idS
Where to find
RTMT, collect “Cisco Identifity
Service” logs
RTMT, same location as IdS
logs, they have the suffix “fedlet-
”
RTMT – Under Cisco Identity
Service you’ll find a folder
”metrics”,
saml_metrics.csv and authorize_
metrics.csv are the relevant
metrics for this document.
Applications/logs for troubleshooting
Application/log Details
Event Viewer in AD FS Allows users to view
the event logs in the system.
Any error in AD FS while
processing the SAML
request/sending the SAML
response will be logged here.
SAML Viewer A SAML Viewer will help in
looking at the SAML Request
and Response that are sent
from/to Cisco IdS.
This browser application is
very useful for the analysis of
SAML Request/Response.
Where to find
In AD FS machine, navigate to Event Viewer
 Applications and Services Logs 
AdDFS 2.0  Admin
These are some suggested SAML viewers
that you can use for looking at the SAML
request and response
Fiddler
How to use fiddler with AD FS
Fiddler Chrome Plugin
SAML Tracer - Firefox
SAML Chrome Panel
Troubleshoot flow diagram
Auth code request processing by IdS
• Link
• Error message:
{"error":"invalid_redirectUri","error_description":"Invalid Redirect URI."}
• Cause:
User accesses application using IP Address/ Alternate Host Name. In SSO mode, if the application is
accessed using IP, it does not work. Applications should be accessed by the hostname by which they
are registered in Cisco IdS. This issue can happen if user accessed an alternate host name that is not
registered with Cisco IdS.
https://uccx1.ccteam.bru.lab:8553/ids/v1/oauth/authorize?redirect_uri=https%3A%2F%2Fuccx-
angelique.ccteam.bru.lab%3A443%2Fappadmin%2Fsso%2Fauthcode&client_id=12bc07eb55dc7495
609bdfdd2e1fa3b083bfeacd&state=aHR0cHM6Ly91Y2N4LWFuZ2VsaXF1ZS5jY3RlYW0uYnJ1LmxhYi
9hcHBhZG1pbi9tYWluCWFwcGxvZ2lu&response_type=code
SAML request initiation by IdS
• Error Message:
{"error":"service_unavailable","error_description":"SAML Metadata is not initialized"}
• Possible Cause:
Idp Metadata is not available in Cisco IdS. Trust establishment between Cisco IdS and AD FS is not
complete.
• Recommended action:
Navigate to Cisco IdS Management console and see if the IdS is in Not Configured state.
Confirm if IdP metadata is uploaded or not.
If not, upload the IdP metadata downloaded from AD FS.
SAML request processing by ADFS
• Error Message:

• Cause:
From ADFS Event viewer log:
Encountered error during federation passive request. Additional Data Exception details:
Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service
URL is not configured. ---> Microsoft.IdentityServer.PolicyModel.Client.PolicyStoreConnectionException:
ADMIN0017: An exception occurred while connecting to the configuration service. The configuration service
URL 'net.tcp://localhost:1500/policy' may be incorrect or the AD FS 2.0 Windows Service is not
running. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to
net.tcp://localhost:1500/policy. The connection attempt lasted for a time span of 00:00:02.0001144. TCP error
code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:1500. ---
> System.Net.Sockets.SocketException: No connection could be made because the target machine actively
refused it 127.0.0.1:1500…
SAML request processing by ADFS #2
• Error Message:
From AD FS Event Viewer

The Federation Service encountered an error while processing the SAML authentication request.

Additional Data

Exception details: Microsoft.IdentityModel.Protocols.XmlSignature.SignatureVerificationFailedException: MSIS0038: SAML

Message has wrong signature. Issuer: 'myuccx.cisco.com'. at
Microsoft.IdentityServer.Protocols.Saml.Contract.SamlContractUtility.CreateSamlMessage(MSISSamlBindingMessage message)
at Microsoft.IdentityServer.Service.SamlProtocol.SamlProtocolService.CreateErrorMessage(CreateErrorMessageRequest
createErrorMessageRequest) at Microsoft.IdentityServer.Service.SamlProtocol.SamlProtocolService.ProcessRequest(Message
requestMessage)

• Possible cause:
Relying party trust is not established or Cisco IdS certificate has changed, but the same is not
uploaded to the AD FS.
SAML request processing by ADFS #2
• Recommended action:
Establish trust between AD FS and Cisco IdS with the latest Cisco IdS certificate.
Ensure that the Cisco IdS Certificate is not expired. You can see the status dashboard in Cisco Identity
Service Management. If so, regenerate the certificate in the Settings page.
SAML Response Sending by ADFS
• Problem:
Browser shows NTLM login, but after successful log in, it fails with many redirects.
• Possible cause:
Cisco IdS supports only form based authentication, Form authentication is not enabled in AD FS.
• Recommended action:
For more details on how to enable Form authentication see:
ADFS 2.0 Form Authentication Setting
ADFS 3.0 Form Authentication Setting
SAML response processing by Cisco IdS
• Error Message:
SAML response processing by Cisco IdS
• ADFS event viewer

• IdS logs:
2017-03-13 21:37:53.189 CET(+0100) [IdSEndPoints-SAML-54] ERROR com.cisco.ccbu.ids IdSSAMLAsyncServlet.java:299
- SAML response processing failed with exception com.sun.identity.saml2.common.SAML2Exception: Invalid Status code in
Response….
SAML response processing by Cisco IdS
• Possible cause:
AD FS is configured to use SHA-256.
• Recommended action:
1. RDP to the AD FS system.
2. Open AD FS console.
3. Select the Relying Party Trust and click Properties
4. Select the Advanced tab.
5. Select SHA-1 from the drop-down list.
SAML response processing by Cisco IdS #2
• Error Message:
SAML response processing by Cisco IdS #2
• ADFS Event Viewer:
SAML response processing by Cisco IdS #2
• IdS logs:
2017-03-13 21:52:50.161 CET(+0100) [IdSEndPoints-SAML-55] INFO com.cisco.ccbu.ids
SAML2SPAdapter.java:76 - SSO failed with code: 1. Response status: <samlp:Status>
<samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Responder">
</samlp:StatusCode>
</samlp:Status> for AuthnRequest: n/a
2017-03-13 21:52:50.161 CET(+0100) [IdSEndPoints-SAML-55] ERROR com.cisco.ccbu.ids
IdSSAMLAsyncServlet.java:299 - SAML response processing failed with exception
com.sun.identity.saml2.common.SAML2Exception: Invalid Status code in Response.
SAML response processing by Cisco IdS #2
• Possible cause:
Custom claim rule is not configured correctly.
• Recommended action:
1. Under AD FS claim rules, ensure that attributes mapping for "user_principal" and "uid" are defined
correctly
2. RDP to AD FS system.
3. Edit the Claim Rules for custom claim rules.
4. Verify that the AD FS and Cisco IdS fully qualified domain names are given.
SAML response processing by Cisco IdS #2
Useful links
• Configure the Identity Provider for UCCX based on SSO
http://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-contact-center-
express/200612-Configure-the-Identity-Provider-for-UCCX.html
• ADFS/IdS Troubleshooting and Common Problems
http://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-contact-center-
express/200662-ADFS-IdS-Troubleshooting-and-Common-Prob.html
• Troubleshooting docwiki
http://docwiki.cisco.com/wiki/Troubleshooting_Tips_for_Unified_CCX_11.5
• Troubleshooting Tech Notes UCCX
http://www.cisco.com/c/en/us/support/customer-collaboration/unified-contact-center-
express/products-tech-notes-list.html
If all of the above fails…
Open a TAC case 
Information that is useful to add from the beginning already:
• Versions
• Topology
• Known changes
• Exact error (with screenshots/logs and your analysis)
• Steps you’ve tried already
• Impact
Questions?