- 637041113

Issue: Not able to make an internal phone call. The error is: High
traffic.
2 remote site are not reachable to each other
-phone 1 >> cucm >> phone 2

- we checked the configs

- performed DNA and route plan report

- it seemed a routing loop issue

- to isolate we configured a pt_test abd assigned it to css_test .

- applied pt_test to the called number

- css_test on to the calling phone and dn both and found that there
is no issue so it was clear now that it's a routing loop issue
- i asked for the traces to rectify this and then mean while you
checked the route pattern report and found 2 same route pattern and
you deleted 1 and issue got fixed

- you asked to monitor the issue till tomorrow and if no CB then

will close the case
626261009

I am receiving the following error message whenever we make a call over

the inter-cluster trunk.
High traffic try again later

On webex:

The problem only happens if we call to a number that doe snot exist on the
other cluster,
then hang up and we call a number that exists.

from 1441 to 6317 the number does not exists

then from 1441 to 6586 high traffic

I took CallManager traces with a range of 5 minutes and there were a lot of
files.

We are also getting route list exhaust error.

>>>

On the traces:

86507049.010 |12:42:16.092 |AppInfo ||PretransformCallingPartyNumber=1441

|CallingPartyNumber=1441
|DialingPartition=Internal

|DialingPattern=XXXX

|FullyQualifiedCalledPartyNumber=6317

86500658.010 |12:42:56.838 |AppInfo ||PretransformCallingPartyNumber=1441

|CallingPartyNumber=1441

|DialingPartition=Internal

|DialingPattern=XXXX

|FullyQualifiedCalledPartyNumber=6586

86500706.002 |12:42:56.845 |AppInfo |In Message -- H225ReleaseCompleteMsg

-- Protocol= H225Protocol

86500706.003 |12:42:56.845 |AppInfo |Ie - Q931CauseIe -- IEData= 08 02 80

AA

86500706.004 |12:42:56.845 |AppInfo |Ie - H225UserUserIe -- IEData= 7E 00

22 05 25 80 06 00 08 91 4A 00 05 11 00 11 00 80 F1 5E 66 B8 FF 51 1B 54 06
A8 01 0A 6A 95 6B 10 80 01 00

86500706.005 |12:42:56.845 |AppInfo |MMan_Id= 0. (iep= 0 dsl= 0 sapi= 0

ces= 0 IpAddr=1ba06a0a IpPort=1720)

86500706.006 |12:42:56.845 |AppInfo |IsdnMsgData1= 08 02 86 54 5A 08 02 80

AA 7E 00 22 05 25 80 06 00 08 91 4A 00 05 11 00 11 00 80 F1 5E 66 B8 FF 51
1B 54 06 A8 01 0A 6A 95 6B 10 80 01 00

86500706.007 |12:42:56.845 |AppInfo |value H323-UserInformation ::=

86500706.008 |12:42:56.845 |AppInfo |{

h323-uu-pdu

h323-message-body releaseComplete :

The disconnection code we are getting is Q931CauseIe -- IEData= 08 02 80 AA

which means "The destination is not reachable because of a temporary
overload on the network switching equipment. Try again later."

Problem Description:
The problem only happens if we call to a number that does not exist on the
other cluster,
then hang up and we call a number that exists.
For example call from CUCM 9 from 1441 to 6317 (this number actually does
not exists on
Cluster with CUCM 7) the number does not exists message comes up.

Then we call from 1441 to 6586 and get high traffic error message.

If we wait 4-5 seconds the issue get resolved.

On traces I found that the call is into a loop.

There is a route pattern "XXXX" on both cluster so when the call is made to
a number that
doe snot exists it is hitting that route pattern and sent back to the
previous cluster.

I hope everything is doing well. I would like to let you know that after
have double checked the logs I found that the call is getting into a loop.
You have a route pattern "XXXX" on cluster with CallManager 9 and also have
a route pattern "XXXX" on cluster with CallManager 7, so when we call to a
number that does not exists in the system we are actually hitting the route
pattern "XXXX" and the call is being sent back to previous cluster and came
into a loop for the route pattern "XXXX".

Please go to the inter cluster trunk and look for the section "Inbound
calls" > Calling search space > and make sure that calling search space
does
not have access to the partition belonging to the route pattern "XXXX".
Please do it for both inter cluster trunk.

We excluded the partition belonging to the RP pointing to the inter-cluster

from the
inter-cluster trunk CSS and that resolved the issue.
\

615254683 good case

I went through the event logs and found the following alert:

Aug 30 09:01:29 IPTUCM2 local7 3 ccm: 2444792: Aug 30 14:01:29.855 UTC :

%CCM_CALLMANAGER-CALLMANAGER-3-ICTCallThrottlingEnd: Cisco CallManager

starts handling
calls for the indicated H323 device which was stopped due to route loop
created over H323

Trunk Device Name.:CCM4_Trunk IP Address:10.100.150.21 Device type.

[Optional]:125 Device

description [Optional].: App ID:Cisco CallManager Cluster

ID:StandAloneCluster Node

ID:IPTUCM2

The errors are an indication of a problem with the gatekeeper trunk

(anonymous device). Call Manager detected a call routing loop over this
trunk so it suspends calls for 30 seconds. This is expected behavior as
there is a timer which governs the amount of time the calls are suspended.
This is called 'H225
Intercluster call throttle timer'.

With the Error: ICTCallThrottlingEnd, Cisco CallManager starts handling

calls for the indicated H.323 device, which is stopped because of the route
loop created over H.323 trunk.

ICT looping can be created by a misconfigured route pattern. This can cause
Cisco CallManager to run high CPU for a long period of time or crash the
server. To solve this problem, Cisco CallManager has added logic in the
H.225 device (for trunk device only) to monitor the number of transit calls
outstanding. Transit call is the call that Cisco CallManager receives the
setup request for (or sends the setup request for) and does not yet receive
or send the first backward message. For example, call proceeding, call
progress, alert, connect, or release complete.

Cisco Call Manager runs a five-second timer to monitor the transit call
queue for the H.225 trunk device. If the number of the transit call queue
entry is greater than a pre-defined threshold, then, for a period of time
(default 30 seconds), all new incoming or outgoing call request to that
H225
trunk device will be rejected by sending release completed message with
cause code Switch System Congestion.

By throttling the new call request when there are unusual high number of
transit calls outstanding, call manager will be able to break the inter
cluster loop, and protecting itself from running high CPU. The 5 seconds
monitor timer is decided based on standard requirement that it will take up
to 4 seconds to receive the first backward message. The predefined
throttling threshold is 140 transit calls/5 seconds per inter cluster
trunk,
which is 28 transit calls/second per inter cluster trunk, which adds up to
100k BHCA per inter cluster trunk. The call throttling period can be
adjusted by a new call manager service
parameter,"TimerH225ICTCallThrottle",
the default value is 30 seconds. When this service parameter is set to 0,
which means there is no call throttling mechanism will be implemented. Test
result shows with 30 seconds of call throttling, the call manager will be
back down to normal CPU in less than one to two minutes.

From CallManager, this is the explanation for the

"TimerH225ICTCallThrottle":
Cisco CallManager can monitor the H225 trunk for an unusually high
number of call attempts, which are caused by events such as
inter-cluster trunk looping. When Cisco CallManager detects an unusually
high number of call attempts for the H225 trunk, it triggers
call-throtling mechanism to reject the new call attempt with switch
system congestion cause code value. By doing so, the Cisco CallManager
can protect itself from high CPU usage or crashing the system. H225
Inter-Cluster Call-Throttle Timer designates the the Cisco CallManager
service parameter that defines how long, in terms of seconds, the Cisco
CallManager will throttle the new call attempt after it detects the
unusually high number of call attempts for the H225 trunk. If the timer
has the value of 0, Cisco CallManager does not throttle new call
attempts. The default value of 30 seconds, which is the result of the
engineering lab test, allows the Cisco CallManager to have reasonable
time to clean up the buffered messages and, at the same time, to allow
the new legitimate call attempt to complete as soon as possible. The
valid values for this field include 0,10, 20, 30, 40, 50, and 60.

The Error: ICTCallThrottlingStart error message indicates that the Cisco

CallManager cannot handle calls for the indicated H.323 device because
of a route loop over the H.323 trunk.

The Error: ICTCallThrottlingEnd error message indicates that the Cisco

CallManager resumed call handling for the indicated H.323 device
(stopped due to the route loop created over the H.323 trunk).

I have read/accepted your service request and would like to retain

ownership

to avoid any delays in resolving your issue.

--

Thanks & Regards

--

Rishabh Marwaha

623287479 Voice mail issue

I have researched on this issue and found that this error occurs when the
inbound CSS for the

ICTs are able to reach partitions containing route patterns that go back
out
this same

ICT.

If this is the case, then we can make a new css for ICT trunk on the call
manager which only have access to

phone and not to route pattern

>>>>

single Unity Connection server with version 8.0.3.20000-18

We do not have any inter cluster trunks.

I have seen the message once my self on a phone and the call flow was the
user trying to
call the voice mail system by pressing the messages button on the phone.
Other than that
I have not seen the error, but they say it happens regularly.

Is this issue isolated to a particular phone model.

Please send me the output from the call manager cli:-

"utils diagnose test"

Do you have any configured VG ports which are not registered currently on
call manager.

Is this error occuring at some point of time in a day. This could be

related
to peak hours load.

I was collecting information from the user and it seems as if it is always

when they call the voicemail system.

Is this happening only for calls which are being forwarded to voice mail
numbers?

Is this an intermittent issue?

Can you disable the G722 codec from phone web page and then make a test
call.

It seems like when the voice mail ports get exhausted, we see a similar
kind
of error on the phones.

While I was analyzing the traces, I saw the following error:-

UNKNOWN_ALARM:HuntListExhausted - HuntListName:PDS_SAA_HL App ID:Cisco

CallManager Cluster ID:StandAloneCluster Node ID:ucmpub|Alarm^*^*

15:00:54.668 |Forwarding - getMaskedDn - maskedDn=[8000] dn=[4001]

mask=[8000]|*^*^*

15:00:54.670 |RouteListControl::idle_CcSetupReq - RouteList(Cisco Unity

Connection Answering Ports), numberSetup=1 numberMember=2
vmEnabled=1|1,100,172,1.1^10.1.100.100^*

15:00:54.671 |CcmCcmdbHelper::buildLineStructGivenLiteDnData -
getCallForwardDynamicRecordGivenNumPlanPkid() FAILED for device
name(CiscoUM1-VI1), numplan pkid(828c0f85-2a4f-638e-7088-cc3b906b370c) -->
IGNORING|*^*^*

Are we using a hunt list to route calls to different voice mail ports? If
yes, please reset the hunt list and then check if any such error occurs
again.

That message would be a call to that building's hunt list and no body
answered and it went to voicemail.

The problem we are seeing is when a user is trying to check there voicemail
from there phone by pressing the message button (2010) and the call is
going
to 8000.

When I setup the system I used the voicemail port wizard in the CM
administration. We have 24 ports configured and when I use the remote port
status monitor I can see that all of the ports are being used in a round
robin fashion. But only 1 or 2 ports are ever used a at time.

Are we using sip trunk to integrate call manager with unity connetions.

Also please check if we have high virtual memory usage on the server.

"show process using-most memory"

"show status"

Please reset the hunt list in off hours and check if the same behavior
re-occurs.

Do send me the RIS data collector perfmon logs from RTMT for all the
servers
to check the memory usage.

Asked him how many ports is he using and how many ports are used at a time
such as "High
Traffic Try Again Later" error occurs. He said that he checked his line
group
configuration and saw that only two voice mail ports are added. He will be
adding other 22
of them in off-hours and will get back to me.
After this the issue is resolved