Debugging Tools

Module Objectives

Upon completion of this module you will be able to

perform the following tasks:

► Troubleshoot AudioCodes’ products using the relevant tools from the variety
of available utilities.

► Correctly report a problem to AudioCodes’ support department

Troubleshooting Guide Lines

► Understanding the problem

► Collecting data
Understanding the Problem

What are the expected results?

What are the actual results?

Collecting Data

When reporting a problem to AudioCodes’ support department

the following information should be provided:

► Test setup (network diagram, call direction, etc.)

► Uploaded ini file
► Syslog trace (without missing messages)
► Unfiltered Wireshark
► Advanced (per request):
► PSTN traces for PSTN problems
► DSP traces for problems related to voice quality,
modem \ fax, DTMF detection, etc.
The ini File

► Must be added to every support request

► Can be downloaded from the Web Interface (Device Actions  Save

Configuration File)

► There is no need to use the Complete ini file; all information can be
obtained from the regular ini file

► Includes information on the software version, networking parameters

and the configuration of the gateway

► When a parameter doesn’t appear in the ini file it indicates that it

received a default value
Syslog
What is Syslog

► Syslog is a standard for forwarding log messages

in an IP network
► A Syslog server should be used to remotely record
logging information
► The Syslog information that is sent by the gateway
is a collection of error, warning and system
messages that record every internal operation of
the gateway
Syslog Syntax (SIP)

► Syslog messages are marked with a sequential

number
► Syslog server usually add the time that the message
was received and the source IP address

18:38:14. 52 : 10.33.45.72 : NOTICE : ( lgr_psbrdex)(619 ) recv <-- DIGIT(0) Ch:0 OnTime:0

InterTime:100 Direction:0 System:1 [File: Line:-1]
18:38:14. 83 : 10.33.45.72 : NOTICE : ( lgr_flow)(620 ) #0:DIGIT_EV [File: Line:-1]
18:38:14. 83 : 10.33.45.72 : NOTICE : ( lgr_flow)(621 ) | #0:DIGIT_EV [File: Line:-1]
18:38:14.958 : 10.33.45.72 : NOTICE : ( lgr_flow)(630 ) | #0:DIGIT_EV [File: Line:-1]

In this example
8 messages were lost
Enabling Syslog

► To enable Syslog from the Web

► Management  Management Settings

[Syslog]
SyslogServerIP = 192.168.2.35
EnableSyslog = 1
SyslogServerPort = 514
Enable SIP Traces

► Always set the debug level to 5 if debug traces are

needed
► Full Configuration Mode  Protocol Configuration  SIP
Advanced Parameters  Advanced Parameters

GwDebugLevel = 5
Enable MGCP / MEGACO Traces

► Full Configuration Mode  Protocol Configuration 

Advanced Configuration

MGCPDebugMode = 3
Online Syslog - Web

► Status & Diagnostics  Message Log

► Doesn’t include the IP address of the gateway
ACSyslog

► ACSyslog is a simple Syslog application that is

provided with the gateway’s software kit
► Doesn’t require installation

Log / stop logging Pause / resume

messages to file Syslog
ACSyslog (cont.)

Location of
Syslog file
ACSyslog (cont.)

Create separate
Syslog files
Syslog Call Example

See Trace Files: Basic MP to MP Call

See Trace Files: Basic ISDN Call

Wireshark
What is Wireshark

► Wireshark is a freeware packet sniffer application

that allows the user to view the traffic that is being
passed over the network

► Used for (live / offline) network troubleshooting and

analysis

► Provides similar functionality to tcpdump but with

GUI interface and many more information sorting
and filtering options.
Q: How do I record the traffic that is sent to / from the
gateway
► Connect the PC and the gateway to the same hub
► Use switch with port mirroring
► Use Debug Recording
Capture Interfaces

► Capture  Interfaces
► Select the network interface that is currently being used by the PC
Capture Options

► Select the ‘Display Options’ and press Start

Main Window

Filter Bar

Packet list pane

Packet details pane

Packet bytes pane

Display Filters

► Display filters allow you to concentrate on the packets you are

interested in while hiding the currently uninteresting ones
► They allow you to select packets by:
► Protocol
► The presence of a field
► The values of fields
► Complex expressions can be used (and &&, or ||, not !)
► Right click on the required field in packet details pane and select
the ‘Apply as Filter’ option

(ip.addr == 10.33.6.100 and sip.Call-ID == "84956707125@10.33.6.100“) and !icmp

Coloring Rules

► Different colors can be assigned for each protocol for better visualization.
► The Coloring list is processed until a match is found, therefore place
general rules (like TCP and UDP) at the bottom of the list
Generating Call Flow

► Visual representation of the entire call flow

► From the Main menu select Statistics  VoIP Calls
Playing G.711 RTP Stream
Analyzing the RTP Data

► Wireshark can be used to analyze the RTP data stream and extract the
audio from the data packets (only for G.711).
► From the Main menu select Statistics  RTP  Stream Analysis
Extracting the Audio

► Save the audio payload of the RTP stream to a file

► Save Payload as a .pcm file
► Select the ‘forward’ option
► Can be played using programs such as CoolEdit
Call Example

See Trace Files: Basic MP to MP Call

See Trace Files: MP to MP Call via TP

Debug Recording
What is Debug Recording

► The Debug Recording (DR) is a mechanism that is used to capture

and record the traffic that is sent / received from / to the gateway:

► Media streams (RTP, T.38 and PCM)

► PSTN messages (ISDN, CAS, SS7)
► Control messages (SIP, MGCP, MEGACO)
► Networking streams (such as HTTP and SCTP)
► Other internal information (such as DSP Events).
The Main Advantages of DR

► Can be used to record all IP traffic that is sent / received from / by

the gateway

► Can be used to record the actual messages that are exchanged

between the gateway and the PBX/PSTN switch

► Can be used to record the actual voice signal that arrives from the
TDM (before it enters the DSP)

► Can be used to record network traffic in environments in which hub /

port mirroring isn’t available

► Can be used to record internal traffic between two endpoints on the

same gateway
Collecting DR Messages

► DR packets are captured using Wireshark

► AudioCodes’ proprietary plugins (supplied in the software kit) must

be placed in the 'plugin' folder of the installed Wireshark version
(typically, c:\Program Files\WireShark\plugins\xxx\, where xxx is the
installed version).

► Use the filter ‘acdr’ (on Wireshark version 0.99.4) to view the DR
messages
DR General Notes

► All PSTN, Media, Control and IP recording can be performed

simultaneously

► All DR rules are deleted after the gateway is reset.

► Save the information that was captured in Wireshark and send it to

AudioCodes support

► DR recording must only be used for debugging purposes. On normal

operation (if not requested differently from AudioCodes’ support) DR
should be disabled.

► To stop the DR, enter the command STOP in the command shell
 Debug Recording

► Lowest level debug capability on the gateway

► Collect all advanced debugging information using a single tool
► Can use switch
► Hub no longer a requirement
► NAT independence
► Collect data from multiple IPs (IP separation)
 What does a Debug Recording Capture?

► PSTN messaging
► TDM audio
► IP audio
► DSP / Host related operations and messaging
► Control Protocol signalling, syslog messaging
► More!
 Debug Recording Trace Points

Info
events
Megaco
MPC MGCP
TPNCP
…
DSP (HPI)
PSTN

TDM Net IP
PCM DSP
Q931 RTP
CAS RTCP
T38
 Where are traces captured?

► Wireshark packet traces will contain the AudioCodes Debug

Recording packets
► AudioCodes supplied Wireshark Plug-in must be placed in a hard
coded directory in order to decode the AudioCodes packets
► Plug-in can be taken from CD handed out in class
 Installing the Plug-in

1. Close Wireshark
2. Open the ‘plugin’ directory for the version of Wireshark currently installed
on the hard drive
► C:\Progarm Files\Wireshark\plugins\0.99.4
3. Unzip or copy the AudioCodes supplied files to the directory above
4. Start Wireshark

► NOTE: The AudioCodes supplied plug-in MUST match the Wireshark

version exactly in order for the application to operate
 Activating the Traces

► CLI
► Recording is done using targets & trace rules which are entered
at the command line on the gateway
► Each trace rule is associated with a target (server)
► Multiple targets and rules are supported
► A target is a pc/laptop running Wireshark with the
AudioCodes plugins
► To capture PSTN signaling, configuration in the ‘Trunk Trace’
page of the web interface is still required
► Access the CLI through telnet or the ‘/FAE’ page in the web
interface
► Change directory from the top level to the ‘DebugRecording’
level to access commands for configuring and executing Debug
Recording trace
 Basic Commands

► Target
► AddIPTarget
► RemoveTarget
► ListTargets
► SetDefaultTarget

► General
► StartRecording
► StopRecording
► RemoveTraceRule
► ListTraceRules
 Working with Targets

► Use ‘ListTargets’ command to show the current list of targets:

/DebugRecording>ListTargets
Available Targets:
0) IP(Host): 192.168.10.51:925 (Default)
► The first target in the list is the default target
► The default port used by the AudioCodes plugins in Wireshark is 925
► Add a target using ‘AddIpTarget’ command. Define the port number to be
used during the recording
/DebugRecording>AddIpTarget 192.168.10.51 925
Available Targets:
0) IP(Host): 192.168.10.51:925 (Default)
 Removing Targets

► Use ‘RemoveTarget’ command to remove a target from the gateway

► A target can only be removed if there are no trace rules pointing to it
► Targets can be removed one by one or all at once
/DebugRecording>RemoveTarget 0
No more targets available
 Trace Rules

► RemoveTraceRule
► ListTraceRules
► AddIPControlTrace
► AddPstnSignalingTrace
► AddNextCallTrace
► AddTrunkBchannelTrace
► AddIPTrafficTrace
► AddChannelIdTrace
 Adding and Removing Trace Rules

► Stop all traces before adding or removing a trace rule

► Rules can be added or removed in any order. The rules are not parsed
hierarchically as they each represent a different trace point on the gateway
► Enter the trace rule without any parameters for syntax help when setting up
the rule
 Control Protocol (SIP) traces

► AddIPControlTrace
► Add IP Control Trace Usage:
► AIPCT TracePoint ControlType [DebugTarget]
► TracePoint - <Net2Host|Host2Net>
► ControlType - <MEGACO|MGCP|TPNCP|SIP>. SIP is recorded
in both trace points
► DebugTarget - defaults to default debug target, see 'ListTargets‘
► Example:
► AIPCT n2h sip
 AddIPControlTrace Trace Points

Info
events
Megaco
MPC MGCP
SIP

DSP (HPI)
PSTN

TDM Net IP
PCM DSP
Q931 RTP
CAS RTCP
T38
 PSTN trace

► AddPstnSignalingTrace
► Add PSTN Signaling Trace Usage
► APST PacketType [DebugTarget]
► Packet Type - <CAS|ISDN|SS7>
► Example:
► APST ISDN
 AddPstnSignalingTrace Trace Points

Info
events
Megaco
MPC MGCP
TPNCP
…
DSP (HPI)
PSTN

TDM Net IP
Q931 DSP
CAS RTP
NO AUDIO RTCP
T38
 AddNextCallTrace for RTP + PCM traffic

► AddNextCallTrace
► Add Next Call Trace Usage:
► ANCT PacketType NumOfCalls [TraceType] [DebugTarget]
► Packet Type - <ALL|ALL-WITH-PCM>. record all media packet
types with or without PCM
► NumOfCalls - amount of next media calls to record
► TraceType - <New|Dynamic>.
► New (defualt) - only the next NumOfCalls new calls will be
recorded.
► Dynamic - active and new calls will be recorded dynamically
until filter is removed.
► DebugTarget - defaults to default debug target, see 'ListTargets’
 AddNextCallTrace Trace Points

Info
events
Megaco
PACKET TYPE MPC MGCP
TPNCP
“ALL” …
DSP (HPI)
PSTN

TDM Net IP
PCM DSP RTP
Q931 RTCP
CAS T38
 AddNextCallTrace Trace Points

Info
events
Megaco
PACKET TYPE MPC MGCP
TPNCP
“ALL-WITH-PCM” …
DSP (HPI)
PSTN

TDM Net IP

PCM
DSP
RTP
RTCP
T38
 Notes on Next Call Trace

► Each time ‘Start’ and ‘Stop’ are toggled, the counter for the next ‘n’ number
of calls defined in the trace rule is reset
► The rule does not have to be re-entered after stopping and starting the
debug recording
 ALL vs ALL-WITH-PCM

► ALL == RTP / IP side audio

► ALL-WITH-PCM = RTP + PCM audio
 Additional Traffic Trace Rules

► AddIPTrafficTrace
► Transport type and\or source\destination

► AddTrunkBchannelTrace
► Record range of channels according to trunk\B Channels

► AddChannelIdTrace
► Record range of channels according to CID

► Use the additional rules when AddNextCallTrace

is not feasible (ie you have to trace a range of
channels on a busy gateway or in very specific
all scenarios)
 Starting and Stopping Traces

Start recording:
=> START
START
recording started

Stop recording:
=> STOP
STOP
recording stopped
 What exactly do I have to do?

1. Log on to the FAE web interface to the gateway (http://<IPofGateway>/FAE) and

select ‘Cmd Shell’
2. Enter ‘DebugRecording’ or ‘DR’ in the command window
3. List all targets using ‘ListTargets’ command
4. Add the IP address and port of the laptop or pc to the list of available targets
5. List the current trace rules with the command ‘ListTraceRules’
6. Determine which CID’s or resources will need to be captured by the traces and use
them to develop the rules in step 7 below.
7. Refer to the two previous slides for the set of rules to add for the debug recording
8. If PSTN tracing is to be utilized, navigate to the ‘Trunk Traces’ link in the FAE page.
Toggle ‘acFullTrace’ for each trunk that will be traced, set tracing from the web to
‘Yes’ and enter the port associated with the IP address of the target entered for the
trace (DIGITAL GATEWAYS ONLY)
9. Enable Syslogs on the gateway and point them to the IP address of the laptop/pc.
Set the verbosity level to ‘5’ for SIP.
10. Start the Wireshark packet trace
11. Enter ‘Start’ at the command line in the web interface to kick off the recording
12. Make the calls and capture the data
13. Stop the Debug Recording in the web interface
14. Stop the Wireshark trace
15. Validate the expected trace points have been captured in the Wireshark trace.
 Working with the traces

► Validate proper data collection

► Control Signaling
► PSTN trace
► IP audio streams
► PCM audio streams
► Extract audio
Control Signaling Trace Rule Validation

Messages will appear

as their protocol type
(ie SIP, MGCP)

Packets have an
AudioCodes Debug
Recording Header in
the UDP packet

Source and Destination of the message are

visible under the ‘Header Extension’ field
PSTN Signaling Trace Rule Validation

Filter by packet type

‘acdr.media_type == 25’ to
display all PSTN trace
packets.

‘NET-BRICKS’ are layer 1

and 2 messages.

‘ACLISDN’ packets contain

the layer 3 Q.931 packets.
It is also possible to simply
filter for ‘q931’ to display
layer 3 messaging only.

All relevant Q.931 data is

contained under the Q.931
header.
 Validating Audio Stream Capture

► Each audio trace (PCM or IP) will contain 2 streams:

► IP:
► DSP  Network (outbound from gateway)
► Network  DSP (inbound to gateway)
► Audio payloads will vary based on selected coder
► PCM:
► DSP  TDM (outbound from gateway)
► TDM  DSP (inbound to gateway)
► PCM Audio will always be either G.711 mu-law or a-law
Viewing Audio Trace Components

Audio packets appear as

RTP (PCM and IP)

Verify the stream for each directoin

of the trace using ‘Trace Point’ field
under ‘AUDIOCODES DEBUG
RECORDING’
 Extracting Audio From Traces

► Follow the same procedures for extracting audio from Debug Recording
traces as for standard network traces using WireShark.
► PCM audio can be saved off and listened to in the appropriate G.711 format
without any modification
► IP audio may have to be converted (i.e. G.729) or otherwise manipulated to
be played back
Chapter 13 – Student Notes