You are on page 1of 103

mcBSC

troubleshooting
Specialist training
Antti Palojrvi

2-9-2014

1
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Agenda

Basics
BCN configuration
BCN flashbank
HiGig
BCN syslog, SEL & Sensors
Debug script

2
1/10/17
Confidential

BSAC
Dpdump & Tcpdump
Megamon
Monitoring external traffic
Traffic flow in mcBSC
mcTC Troubleshooting

Nokia 2014 - File Name - Version - Creator - DocID

Basics
Connection to LMP 1/2
Connection to LMP can be taken locally via serial or Ethernet mgmt port.
From BSC connection is taken with telnet by using Virtual MML in OMU.
Connection is taken to the BCNs VLAN800 address
Connection to MCBC-0
00-MAN> ZLP:1,VIM
00-MAN> 1CT::10.10.10.10
VIMMLA: connecting to: 10.10.10.10
VIMMLA: connected to: 10.10.10.10
Linux 2.6.21.7-hrt1-WR2.0ap_standard (BCNMB-A)
(09:47 on Wednesday, 21 May 2014)
login: root
Password:
Last login: Wed May 21 08:58:18 from 10.144.209.2
Wind River Linux glibc_small (standard) 2.0
root@BCNMB-A:~#
3
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Default IP addresses
10.10.10.10 for box1
10.10.10.11 for box2
10.10.10.12 for box3
10.10.10.13 for box4
10.10.10.14 for box5
10.10.10.15 for box6
10.10.10.16 for box7

Basics
Connection to LMP 2/2
Common problems in LMP connection.
Session via VIMLA fails
00-MAN> 1CT::10.10.10.10
VIMMLA: connecting to: 10.10.10.10
VIMMLA: connect() error: 36BC
00-VIM>
Session from other BCN fails
root@BCNMB-A:~# telnet 10.10.10.10
Trying 10.10.10.10...
telnet: connect to address 10.10.10.10: Connection
refused
root@BCNMB-A:~#
Possible reason: telnet not activated in BCN
Resolution: root@BCNMB-A:~# chkconfig telnet on

Session via Ethernet mgmt port fails.


Check that the mgmt ifeth configuration is in order. For example node 3
root@BCNMB-B:~# ifconfig eth0
eth0
Link encap:Ethernet HWaddr 00:D0:C9:DA:ED:F4
inet addr:192.168.0.12 Bcast:192.168.0.255 Mask:255.255.255.0
Actual configuration is defined in network-services
root@BCNMB-A:~# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.12
#GATEWAY=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
Once configuration is in order restart the network services
root@BCNMB-A:~# service network restart
!Note that the restart has affect also to vlan800 so before restart it has to be absolutely sure that
configurations are in order. Refer to the commissioning documentation.

4
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Basics
Connection to BCN switches 1/2
Connection to BCN switches can be taken by activating screen application in
LMP or with telnet via LDI.
Screen from LMP:

Telnet:

root@BCNMB-A:~# screen -r
User:admin
Password:
(KoffM1MainSwitch) >

ZW6T:MCBC,100:;

!Do not use screen without any parameter

(KoffM1MainSwitch) >

Screen has sessions open to FASTPATH switches and MCH_CLI.


Transfer between these:
Ctrl-g + 0,1 between switches
Ctrl-g + 2 to MCH_CLI
Ctrl-g + d detach the screen back to the original console

Session closed with ctrl-c

5
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Telnet connection successfully opened to MCBC 100 (BCN_A),


IP=10.10.10.10.

Basics
Connection to BCN switches 2/2
Common problems in connecting to switches
Screen from LMP fails:

Telnet fails:

root@BCNMB-A:~# screen -r
There is a screen on:
3661.console.BCNMB-A
(Attached)
There is no screen to be resumed.
root@BCNMB-A:~#

ZW6T:MCBC,100:;
LOADING PROGRAM VERSION 6.19-0

Possible reason:
Someone is using screen from another session.
Screen session is not detached before closing terminal session.
Resolution:
Make sure that no-one else is connected and applying changes to the
configuration.
Detach the screen
root@BCNMB-A:~# screen -d
[3661.console.BCNMB-A detached.]

6
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

/*** DX ERROR: 16492 ***/


/*** COULD NOT CONNECT TO TELNET SERVER ***/
Possible reason : Vlan800 is not working in BCN.
Resolution: Take local session to BCN and check that vlan configuration file is
in order
root@BCNMB-A:~# cat /etc/sysconfig/network-scripts/ifcfg-eth0.800
DEVICE=eth0.800
VLAN=yes
BOOTPROTO=static
ONBOOT=yes
PERSISTENT_DHCLIENT=Y
DHCLIENTARGS="-nw"
BROADCAST=10.10.10.255
IPADDR=10.10.10.10
NETMASK=255.255.255.0

Basics
Connection to BCN add-in cards 1/2
Connection to BCN add-in cards is taken with minicom from LMP.
Needed for example while collecting the restart logs from the unit.
root@BCNMB-A:~# minicom cpu5
Welcome to minicom 2.2
OPTIONS: I18n
Compiled on Dec
Port /dev/ttyS2

4 2013, 11:18:00.
Press CTRL-A Z for help on special keys

Normal Image size 0x0005e6e8 crc 0xbf4ecb1e


U-Boot 1.1.1 SW 3.1.3 (Build time: Jun 20 2013 - 12:37:42)
OCTEON CN5650-NSP pass 2.1, Core clock: 800 MHz, DDR clock: 330 MHz (660 Mhz dat
DRAM: 8192 MB
Flash: 256 MB
Clearing DRAM........ Done
BIST check passed.

7
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Basics
Connection to BCN add-in cards 2/2
Common problems in connecting to BCN add-in card
Minicom session fails
root@BCNMB-A:~# minicom cpu5
Device /dev/ttyS2 is locked.
root@BCNMB-A:~#
Possible reason :
Minicom session is in use or stucked.
Resolution:
Make sure that no-one else is connected.
Check and kill the minicom process
root@BCNMB-A:~# ps aux | grep "minicom"
root
14010 0.0 0.2
3472 2156 ttyS0
root
15417 0.0 0.0
2044
480 pts/7
root@BCNMB-A:~#
root@BCNMB-A:~# kill -9 14010
root@BCNMB-A:~# ps aux | grep "minicom"
root
16674 0.0 0.0
2044
480 pts/7
root@BCNMB-A:~#

8
1/10/17
Confidential

S+
R+

14:01
14:02

0:00 minicom cpu5


0:00 grep minicom

R+

14:03

0:00 grep minicom

Nokia 2014 - File Name - Version - Creator - DocID

Basics
File transfer between BSC and BCN module (LMP) 1/2
Files can be transferred with FTP between BSC file system and LMP
FTP connection in LMP is taken into OMUs VLAN 800 address.
UNIT
INTERFACE
-----------OMU
EL0

ADMIN
ADDR
STATE MTU
ATTR
IP ADDRESS
----- ----- -------- --------------UP

1500

EL1

UP

1500

VLAN20

UP

1500

->EL0

20

VLAN800

UP

10.0.1.1/24

LI

10.10.10.1/24

0/0
1500

If file name is too long it can be change with move cmd.

->EL0
800 0/0
root@BCNMB-A:/tmp# mv debug_65_2_G7112405276.log DEBUG1.LOG
VLAN3910

UP

1500

10.9.192.236/29

Sometimes default storage is not enough for the uploaded file.


It can
be extended
in 0/0
FTP session with quote allo command
->EL1
3910
ftp> quote allo 99999

9
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Basics
File transfer between BSC and BCN module (LMP) 2/2
Common problems in FTP between BSC file system and LMP
FTP from LMP to OMU fails
root@BCNMB-A:~# ftp 10.10.10.1
Connected to 10.10.10.1.
220 Service ready for new user
Name (10.10.10.1:root): SYSTEM
331 Username ok, need password.
Password:
530 Log in incorrect.
ftp: Login failed.
Root cause:
FTP not in use
Reolution:
Enable FTP server with MML ZIAA:PROFILE::::FTP=W:;

10
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Basics
Esw version info in LMP side.
sw_fw_versioninfo command is used to check the esw of the BCN and add-in
cards
in LMP.
root@BCNMB-A:~# sw_fw_versioninfo
Active U-Boot Version
Backup U-Boot Version
LMP Version
PCB Version
LED CPLD Version
PCI-LPC bridge XP2 Version
VCMC Version
PWR1014 Version
FRUD Version
Part Number

11
1/10/17
Confidential

3.1.7 (in flash 0)


3.1.7 (in flash 1)
3.1.7
A103-2/A103-3/A103-4/A103-5/A103-6/A103-7
05
05
3.1.6
0007
3.1.7
C111721.A1A

Nokia 2014 - File Name - Version - Creator - DocID

BCN configuration

Faults in BCN configuration can cause problems to MCBC


management, unit restarts & disc access.

12
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN configuration
Management VLAN800

Management VLAN is essential for the LDI, SNMP, RMCP & ESW mgmt.
By default LMP has static IP for VLAN800.
LMP VLAN800
root@BCNMB-A:~# cat /etc/sysconfig/network-scripts/ifcfg-eth0.800
DEVICE=eth0.800
VLAN=yes
BOOTPROTO=static
ONBOOT=yes
PERSISTENT_DHCLIENT=Y
DHCLIENTARGS="-nw"
BROADCAST=10.10.10.255
IPADDR=10.10.10.10
NETMASK=255.255.255.0
NSN factory default is to define static IP for LMP in VLAN800 instead of DHCP.
In case IP definitions are not in order various problems will be faced. LDI does
not work, telnet does not work, SNMP traps do not work etc.
BSAC VLAN800
BSAC uses DHCP to get the VLAN800 IP address from OMU.

13
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

VLAN800 configuration file


Addresses that the OMU assigns are defined in ILLHKDGX.XML in LFILES
directory
For example PTUM-0
<!-- PTUM-0 -->
<client
l_cli_name
= "PTUM-0"
l_cli_ip
= "10.10.10.50"
l_cli_mask
= "255.255.255.0"
l_cli_mask_len
= "24"
l_cli_boot
= "">
</client>
Detailed troubleshooting can be done with DPDUMP from OMU

BCN configuration

Node and Rack definition


BCN contains configurations related to rack and node numbers which need to
be in line with the HW configuration.
MCBC-0
root@BCNMB-A:~# mch_cli GetNodeNumber
1
root@BCNMB-A:~# mch_cli GetRackNumber
65
root@BCNMB-A:~#
MCBC-1
root@BCNMB-A:~# mch_cli GetNodeNumber
2
root@BCNMB-A:~# mch_cli GetRackNumber
65
root@BCNMB-A:~#

WTI:P:MCBC,;
EXECUTION STARTED
READING DATA FROM DATABASE ...
MCBC-0
BCN_A

IN LOC 1A001-00
0 TRACK: 1

MODULE TYPE: SHM

MODULE NUMBER:

MCBC-1
BCN_A

IN LOC 1A002-00
0 TRACK: 1

MODULE TYPE: SHM

MODULE NUMBER:

TOTAL OF

UNITS,

PLUG-IN UNITS AND

FUNCTIONS

If these settings are not in order the units in the BCN are not restarting
properly since DHCP assigns IP address based on rack and node definitions.
14
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN configuration

Boot parameters & card configuration 1/3


Booting order, boot script and card configuration of the units within BCN is
defined in /etc/ipmi/mch.conf
Boot order
boot_order=8 1 2 3 4 5 6 7 (MCBC-0)
boot_order=1 2 3 4 5 6 7 8 (MCBC-1->7)
In MCBC-0 OMU booted first. In other modules boot starts from cpu 1
Boot script & card config
######################
# CPU Add-in Card 8
######################
8.bootscript=99
8.enable=1
8.card_conf=0x88 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
For all the BJC-A cards the boot script is 99 and for BOC-A 17.
First byte of the card config shall be:
0x88 for the OMU
0x08 for the MCMUs and BCXUs
0x18 for the BOC-A units

If boot script is not in order the unit does not restart properly.
First byte of the card_conf defines the root complex, used synch source and
CPU
clock Nokia
speed.
15
1/10/17
2014 - File Name - Version - Creator - DocID
Confidential

BCN configuration

Boot parameters & card configuration 2/3


The Node, Rack and boot script that BCN assigns to the BOC-A unit can be
seen in the restart log of the unit.
ETME restart log:
MMC: Card Config
pci_host_mode 0 pci_boot 0 fail_safe 0 octeon_speed 8 clock_source 0
boot_flash 0 buffer_dir 0 rack 65 node_number 2 slot_number 7 boot_script 17
### mmc bootcmd="run bootscript_17
Hit key 'u' to stop autoboot: 3
Interface 0 has 1 ports (XAUI)
Interface 1 has 1 ports (XAUI)
Interface 2 has 4 ports (NPI)
Interface 3 has 4 ports (LOOP)
BOOTP broadcast 1
octeth0: Down (port 0)
octeth0: Up 10000 Mbps Full duplex (port 0)
BOOTP broadcast 2
DHCP client bound to address 10.0.1.34
Using octeth0 device
TFTP from server 10.0.1.11; our IP address is 10.0.1.34
Filename '/LFILES/METESCRA.IMG'.

16
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN configuration

Boot parameters & card configuration 3/3


The card config in the mch.conf file is the one the BCN uses during restart.
Actual card config is stored in add-in card FRU data which can be read with
BOC-A
mch_cli
root@BCNMB-A:~# mch_cli GetCardConfig cpu5
PCI host mode: EP
PCI boot: no
Failsafe: no
clock source: external
Octeon speed: 800MHz
Boot flash: 0
Buffer dir: 0
other config:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
hwargs: 0x18 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00
BCJ-A (OMU)
root@BCNMB-A:~# mch_cli GetCardConfig cpu8
PCI host mode: RC
PCI boot: no
Failsafe: no
clock source: internal
Octeon speed: 800MHz
Boot flash: 0
Buffer dir: 0
other config:01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
hwargs: 0x88 0x00 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00
17
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN configuration

Root complex parameters


In mcBSC root complex is defined to OMU
If OMU boots as a end point it can not find the disk
OMU boot with wrong root complex definition.
DMX FPGA Information
FPGA Code: 061bW VARCODE[0:1] BJC-A
CPLD Version: 0x8f
Warm Reset Counter: 5
Booted from Flash: SPI0
Bootscript: [Boot device = 1, Subdevice = 1, Disabled]
Setup Information: Modified
Reset Type: Cold Reset
ME Firmware: 6.0.30.1199
Mass Storage Detected
Add-in Card booting as Endpoint (EP)
UNIT = 0000, ACTIVE
DISK DRIVE SYSTEM NOT OPENED: 00000033
CHOOSING PACKET
FAILED: DRIVE(S) NOT FOUND: W0,W1
EMB-0 AT ETH PORT 01: LINK OK, 1000 Mbit/S
EMB-1 AT ETH PORT 02: LINK OK, 1000 Mbit/S
REGISTRATION TO COMPUTER 0005 OK
LOADING NET CONFIGURATION FROM COMPUTER 0001
EMB STATUS = 00004004
LOADING NOT SUCCEEDED, STATUS = 00004004

18
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Root complex definitions are in FRU data


MCBC-0
root@BCNMB-A:~# mch_cli GetPCIeRC
RC: 8
root@BCNMB-A:~# mch_cli GetFRUPCIeRC
Slot8
root@BCNMB-A:~#
Other MCBC units
root@BCNMB-A:~# mch_cli GetPCIeRC
RC: 0
root@BCNMB-A:~# mch_cli GetFRUPCIeRC
LMP
root@BCNMB-A:~#
Modification
root@BCNMB-A:~# mch_cli SetPCIeRC <8|0>
root@BCNMB-A:~# mch_cli SetFRUPCIeRC <slot8|LMP>

BCN configuration

Message bus address definitions


Message bus address is updated to BCN FRU when the add-in card is created to
the HW database.
Therefore it is essential that the VLAN800 is working in every BCN while HW is
created.
MB addresses can be checked
from
FRU dataWith
Check unit addres
with BCN
IIM

Check MB addresses in LMP


mch_cli FRU info 0x20

[NSN.EMB Addresses]
Vendor Format Version = 0
Slot 1 = 0x0004
Slot 2 = 0x0049
Slot 3 = 0x004A
Slot 4 = 0x1400
Slot 5 = 0x1401
Slot 6 = 0x1402
Slot 7 = 0x1403
Slot 8 = 0x0000

0x20 == LMP IPMI address

19
1/10/17
Confidential

ZDDS;
00-MAN> ZLP:1,IIM;
00-MAN> Z1RREU:12C,0:100;
emb_addr: 1403
Completion code: 00
Command Completed Normally.
Syntax:
Z1RREU:<nit_type>,<unit_index>:100;
Unit types:
etme_c = 128
pcum_c = 129
etma_c = 12C
mcmu_c = 6A
bcxu_c = 12B

Nokia 2014 - File Name - Version - Creator - DocID

IIM the MB adress can also be updated if needed


ZDDS;
00-MAN> ZLP:1,IIM;
00-MAN> Z1RWEU:12C,1:100:1408
Completion code: 00
Command Completed Normally.
Syntax:
Z1RWEU:<unit type>,<unit index>:100:<emb address>;

BCN configuration
IPMI LAN configuration

IPMI LAN configuration has to be in order so that the functionalities based on


SNMP traps work properly.
- External alarms, LDI supervision & BCN HW related alarms.
root@BCNMB-A:~# cat /etc/ipmi/ipmi_lan.conf
#addr 10.10.10.10 623
priv_limit admin
allowed_auths_callback none md5
allowed_auths_user none md5
allowed_auths_operator none md5
allowed_auths_admin none md5
user 2 on "admin" "admin" admin 5 none md5
# destination <number> <ack|noack> <type> <timeout> <retry> <ip>
destination 1 noack 0 10 3 10.10.10.2
Destination has to point MCMU vlan800 address.
destination <number> <ack|noack> <type> <timeout> <retry> <ip>
Type: 000b = Pet Trap destination
20
1/10/17
Nokia 2014 - File Name - Version - Creator - DocID
Confidential

BCN configuration
Reset scripts

In each BCN of the mcBSC there are reset scripts running which control the unit
restarts during LMP restart.
These are nodeReset.sh and resetNodes.sh
root@BCNMB-A:~# ls -la /etc/rc.d/init.d/resetNodes.sh
-rwxr-xr-x 1 root root 707 Jul 16 10:57 /etc/rc.d/init.d/resetNodes.sh
root@BCNMB-A:~# ls -la /usr/bin/nodeReset.sh
-rwxr-xr-x 1 root root 2883 Jul 16 10:57 /usr/bin/nodeReset.sh
root@BCNMB-A:~# chkconfig --list | grep "reset"
resetNodes.sh
0:off
1:off
2:off
3:on
4:off
root@BCNMB-A:~#

5:off

6:off

If these are not in order units will not restart properly during MCBC restart.

For detailed installation instructions refer to the esw installation instructions in


release documentation.
21
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Flash bank

BCN, BOC-A, BMPP2 and BSAC have two separate flash bank
with own esw and configuration (BCN). Misalignment of
these can cause various problems due esw incompatibility
or missing configuration in case flash bank is changed.

22
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN flashbank
Flashbank basics

BCN has two separate flash banks for esw binaries and non FRU related
configurations
Esw binaries are separately stored and updated in flash banks
BCN flash bank specific configuration
Ethernet interface configurations
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-eth0.800
Add-in card related configuration in /etc/ipmi/mch.conf
HiGig configurations
Switch configurations

Since the Switch configurations are flash bank specific it is important to update
the back-up bank as well when switch configuration is modified. This can be
done either by changing the flash bank and doing the configuration again or
applying the SWTN 910 which is recommended.
23
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN flashbank
Esw update

When esw of the BCN is updated with ZWD commands back-up bank is always
the one which is updated. Therefore possible failures in the update procedure do
not necessarily cause immediate affect to the system.
Current active bank can be checked with interrogate command or from BCN
LMP with mch_cli.
ZWDT:UT=MCBC,UI=1,;
1 VALID PLUG-IN UNITS FOUND

root@BCNMB-A:~# mch_cli GetBootFlashBank 0x20


Boot flash bank: 0
root@BCNMB-A:~#

EXECUTION STARTED AT 2014-06-16 13:09:53


0% DONE, PLEASE WAIT
0 H,
2 MIN AND
100% DONE, PLEASE WAIT
0 H,
0 MIN AND
FUNCTIONAL
UNIT

PLUG-IN UNIT
TYPE INDEX

----------MCBC-1

------------BCN_A-0

24
1/10/17
Confidential

20 SEC
0 SEC

CURRENT ACTIVE
CURRENT BACK-UP
DISK FILE
UPDATE STATUS
----------------------------------BNCMBBZZ.PAC 1.7-8 13/12/13
F706C56B
BNCMBBZZ.PAC 1.7-8 13/12/13
F706C56B
BNCMBBZZ.PAC 1.7-8 13/12/13
F706C56B
UPDATED (ACTIVE FLASH BANK 0)

Nokia 2014 - File Name - Version - Creator - DocID

0x20 == LMPs IPMI address

BCN flashbank
Esw update

For a successful update FTP and telnet has to be in use in BCN.


Example where telnet does not work in LMP.
ZWDT:UT=MCBC,UI=1,;
1 VALID PLUG-IN UNITS FOUND
EXECUTION STARTED AT 2014-08-29 13:04:43
0% DONE, PLEASE WAIT
0 H,
2 MIN AND
100% DONE, PLEASE WAIT
0 H,
0 MIN AND
FUNCTIONAL
UNIT

PLUG-IN UNIT
TYPE INDEX

----------MCBC-1

------------BCN_A-0

25
1/10/17
Confidential

20 SEC
0 SEC

CURRENT ACTIVE
CURRENT BACK-UP
DISK FILE
UPDATE STATUS
----------------------------------VERSION INQUIRY FAILED
VERSION INQUIRY FAILED
BNCMBBZZ.PAC 1.7-8 13/12/13
F706C56B
BOOT STATUS INQUIRY FAILED

Nokia 2014 - File Name - Version - Creator - DocID

BCN flashbank
Esw update

After successful update unit starts to wait reboot to activate current (newly
updated) back-up bank.
ZWDT:UT=MCBC,UI=1,;
1 VALID PLUG-IN UNITS FOUND
EXECUTION STARTED AT 2014-06-16 13:09:53
0% DONE, PLEASE WAIT
0 H,
2 MIN AND
100% DONE, PLEASE WAIT
0 H,
0 MIN AND
FUNCTIONAL
UNIT

PLUG-IN UNIT
TYPE INDEX

----------MCBC-1

------------BCN_A-0

20 SEC
0 SEC

CURRENT ACTIVE
CURRENT BACK-UP
DISK FILE
UPDATE STATUS
----------------------------------BNCMBBZZ.PAC 1.7-4 13/04/22
3FD5BCB6
BNCMBBZZ.PAC 1.7-4 13/04/22
3FD5BCB6
REBOOT PENDING (ACTIVE FLASH BANK 0)
NEXT REBOOT WILL ACTIVATE BANK 1

Reboot:
In such a state BCN unit can be rebooted from MML with
ZWDU:UT=MCBC,UI=1,DO=REBOOT:; (NOTE! Unit shall be in TE state)
BCN can be rebooted also from LMP:
root@BCNMB-A:~# reboot
26
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN flashbank
Flashbank change

In case BCN back-up flash bank need to be activated due misaligned esw
version or due BCN configuration change it can be done in u-boot by interrupting
the BCN restart.
Activity can only be performed via serial interface to the BCN
Note that this has a affect to the system since BCN and units in it will be
restarted.

root@BCNMB-A:~# reboot
INIT: Sending processes the TERM signal
Shutting down sfpd: [ OK ]
.
.
.
Hit key 'u' to stop autoboot: 3
//at this point there is 3sec time to press u button
Now the flash selection byte can be changed with ipmi raw command
Activate flash 1
BCNMB-A# ipmi raw 2e 04 2a 6f 00 00 00 80 00 00
Activate flash 0
BCNMB-A# ipmi raw 2e 04 2a 6f 00 00 00 00 00 00
After flash selection byte is set BCN can be rebooted
BCNMB-A# reset

27
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BOC-A, BMPP2 and BSAC flashbank


Flashbank basics

BOC-A, BMPP2 and BSAC have two separate flash banks also for the esw
binaries.
Active flash bank can be checked from MML and LMP similar way as for the
BCN.
root@BCNMB-A:~# mch_cli GetBootFlashBank 0x7e
Boot flash bank: 1
root@BCNMB-A:~#

ZWDT:UT=ETME,UI=1;
LOADING PROGRAM VERSION 13.20-0
EXECUTION STARTED
1 VALID PLUG-IN UNITS FOUND

EXECUTION STARTED AT 2014-06-16 14:29:57


0% DONE, PLEASE WAIT
0 H,
2 MIN AND
100% DONE, PLEASE WAIT
0 H,
0 MIN AND
FUNCTIONAL
UNIT

PLUG-IN UNIT
TYPE INDEX

----------ETME-1

------------BOC_A-0

28
1/10/17
Confidential

IPMI addresses of the add-in cards.


1 = 0x72, 2 = 0x74, 3 = 0x76, 4 = 0x78, 5 = 0x7A, 6 = 0x7C
7 = 0x7E, 8 = 0x80
20 SEC
0 SEC

CURRENT ACTIVE
CURRENT BACK-UP
DISK FILE
UPDATE STATUS
----------------------------------BNCOCBZZ.PAC 1.2-3 13/06/28
535EBAFE
BNCOCBZZ.PAC 1.2-3 13/06/28
535EBAFE
BNCOCBZZ.PAC 1.2-3 13/06/28
5376FAFE
UPDATED (ACTIVE FLASH BANK 1)

Nokia 2014 - File Name - Version - Creator - DocID

Cpu id can be also used as a parameter


root@BCNMB-A:~# mch_cli GetBootFlashBank cpu7
Boot flash bank: 1
root@BCNMB-A:~#
BSAC flashbank is checked with ipmitool
root@BCNMB-A:~# ipmitool -t 0x84 -b 7 raw 0x2e 0x1b 0x2a 0x6f 0x00
2a 6f 00 00
root@BCNMB-A:~#
Last byte from the response indicates the active bank.

BOC-A, BMPP2 and BSAC flashbank


Flashbank change

If BOC-A or BSAC flash bank need to be changed it can be done via BCN LMP
BOC-A
Via minicom session enter into the BOC-A u-boot by interrupting the BOC-A boot same way as the BCN.
Octeon Reset Reason: Commanded Reset
MMC: Card Config
pci_host_mode 0 pci_boot 0 fail_safe 0 octeon_speed 8 clock_source 0
boot_flash 0 buffer_dir 0 rack 65 node_number 2 slot_number 7 boot_script 17 mad
### mmc bootcmd="run bootscript_17"
Hit key 'u' to stop autoboot: 3
//at this point there is 3sec time to press u button
Now the BOC-A flash bank selection can be changed with IPMI command
Activate Flash bank 1
ipmi [B8 00 04 2A 6F 00 00 08 80 00 00]
Activate Flash bank 0
ipmi [B8 00 04 2A 6F 00 00 08 00 00 00]
After flash selection byte is set BOC-A can be rebooted
Octeon BCNOC-A# reset

29
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BOC-A, BMPP2 and BSAC flashbank


Flashbank change

BSAC AMC slot has to be deactivated while the flash bank is being changed.
BSAC
root@BCNMB-A:~# mch_cli Deactivate amc2
Set fru activation policy: clear deactivation lock bit!
deactivate: Command completed normally
Now the BOC-A flash bank selection can be changed with IPMI command
Activate Flash bank 1
root@BCNMB-A:~# ipmitool -t 0x84 -b 7 raw 0x2e 0x1C 0x2a 0x6f 0x00 0x01
Activate Flash bank 0
root@BCNMB-A:~# ipmitool -t 0x84 -b 7 raw 0x2e 0x1C 0x2a 0x6f 0x00 0x00
root@BCNMB-A:~# mch_cli Activate amc2
Clear locked bit!
done: Command completed normally
Set FRU Activation!
done: Command completed normally
root@BCNMB-A:~#

30
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

HiGig

HiGig creates the loop free LAN between the BCNs in the
mcBSC.
HiGig has to work properly the ensure traffic between the
BCN modules.

31
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

HiGig
HiGig configuration issues can be investigated from LMP.
HiGig is configured separately in BCNs therefore need to be checked from
each
BCN.
Symbolic
link /mnt/fastpath/module should be correctly defined based on node number. Node 1: m10,
Node 2 m20
root@BCNMB-A:~# ls -la /mnt/fastpath/module
lrwxrwxrwx 1 root root 16 Feb 19 20:26 /mnt/fastpath/module -> /opt/cluster/m10
root@BCNMB-A:~#
Behind the link the init.soc must have proper configuration.
root@BCNMB-A:~# cat /mnt/fastpath/module/init.soc
echo
echo init mesh6 ver 0.3.9, module 10
0:port 0x1fffffff en=f
0:local module 10
0:mod port 0 29 my_modid=@module
0:mod modport_map 0x0 7424 higig_port_bitmap=0x00000000
0:mod src_modid_block 0x1500 256 port_block_mask_bitmap=0x1fffffff
0:mod src_modid_block 0x1600 256 port_block_mask_bitmap=0x1fffffff
0:mod src_modid_block 0x1700 256 port_block_mask_bitmap=0x1fffffff
0:mod src_modid_block 0x1800 256 port_block_mask_bitmap=0x1fffffff
0:mod src_modid_block 0x0100 256 port_block_mask_bitmap=0x1fffffff
root@BCNMB-A:~#
In BCN-A environment the used mesh files are mesh6 and mesh7.
- mesh6 in case 10G site connectivity is used
- mesh7 in case 10G site connectivity is not used. (this is factory default config)
32
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

HiGig
HiGig port state changes are written into the screen console if the state
changes.
Alarm
isprompted
alsotoraised
if there is a failure in interconnection cable.
HiGig state3292
changes are
the screen terminal.
m10 SFP+1 (phy21) DOWN, set secondary via SFP+2 (phy22)
m10 SFP+1 (phy21) UP
<HIST> mcKOFF
OMU
** ALARM ........
(0070) 3292 INTERNAL LAN LINK BROKEN

TRANSM
2014-06-11
ZYDEMI

14:45:30.04

MCBC 100d BCN_A 0d 0015 MCBC 101d BCN_A 0d 0015 255d

HiGig
(KoffM1MainSwitch)
#show port
all
Admin mode:
port admin
mode
and link status can be checked
from main switch
Admin
Physical
Physical
Intf
Type
Mode
Mode
Status
------ ------ ------- ---------- ---------0/1
Enable
0/2
Enable
0/21
Enable 10G Full
10G Full
0/22
Enable 10G Full
10G Full
0/23
Enable
0/24
Enable
33
1/10/17
Confidential

Link
Status
-----Down
Down
Up
Up
Down
Down

Link
Trap
------Enable
Enable
Enable
Enable
Enable
Enable

LACP
Mode
-----Enable
Enable
Enable
Enable
Enable
Enable

Nokia 2014 - File Name - Version - Creator - DocID

Actor
Timeout
-------long
long
long
long
long
long

Tells if the port is shutdown in the


configuration.

Link status mode:


Tells info about physical connection.

HiGig
HiGig port specific config files are located in /mnt/fastpath/module/u0/ and all
the HiGig ports need phy<port>down.soc and phy<port>up.soc files.
mesh 7 environment:
root@BCNMB-A:~# ls -la
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root
-rw-r--r-- 1 root root

/mnt/fastpath/module/u0/phy*
1854 Jul 16 10:57 /mnt/fastpath/module/u0/phy1down.soc
1819 Jul 16 10:57 /mnt/fastpath/module/u0/phy1up.soc
1855 Jul 16 10:57 /mnt/fastpath/module/u0/phy21down.soc
1820 Jul 16 10:57 /mnt/fastpath/module/u0/phy21up.soc
1855 Jul 16 10:57 /mnt/fastpath/module/u0/phy22down.soc
1820 Jul 16 10:57 /mnt/fastpath/module/u0/phy22up.soc
1855 Jul 16 10:57 /mnt/fastpath/module/u0/phy23down.soc
1820 Jul 16 10:57 /mnt/fastpath/module/u0/phy23up.soc
1855 Jul 16 10:57 /mnt/fastpath/module/u0/phy24down.soc
1820 Jul 16 10:57 /mnt/fastpath/module/u0/phy24up.soc
1854 Jul 16 10:57 /mnt/fastpath/module/u0/phy2down.soc
1819 Jul 16 10:57 /mnt/fastpath/module/u0/phy2up.soc

If these are missing or corrupted the HiGig does not work properly.

34
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN log & Sensors

BCN logs are stored in syslog and System Event Log like in
any other Linux system.
BCN has also various sensors that can be investigated from
BCN LMP.

35
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BCN syslog
System log of the Linux running on BCN is located in /var/log/syslog as in any
other Linux system.
How to print tail
root@BCNMB-A:~#
Jun 16 14:59:47
Jun 16 15:00:01
Jun 16 16:03:47
Jun 16 16:03:47
Jun 16 16:04:02
Jun 16 16:04:02

of the syslog constantly


tail -f /var/log/syslog
BCNMB-A xinetd[2697]: EXIT: telnet status=0 pid=1241 duration=59(sec)
BCNMB-A crond[3063]: (root) CMD (/usr/lib/sa/sa1 1 1)
BCNMB-A mch: event 0x0134 match channel 7, send to dest 1
BCNMB-A mch: send trap -> event=0x0134, ip=10.10.10.2
BCNMB-A mch: event 0x0135 match channel 7, send to dest 1
BCNMB-A mch: send trap -> event=0x0135, ip=10.10.10.2

Command can be interrupted with ctrl+c


Syslog can be investigated like in any other linux system
root@BCNMB-A:~# cat /var/log/syslog | grep "telnet
Jun 16 14:56:50 BCNMB-A xinetd[2697]: START: telnet pid=30901 from=10.10.10.2
Jun 16 14:57:49 BCNMB-A xinetd[2697]: START: telnet pid=32298 from=10.10.10.2
Jun 16 14:57:49 BCNMB-A xinetd[2697]: EXIT: telnet status=0 pid=30901 duration=59(sec)
Jun 16 14:58:48 BCNMB-A xinetd[2697]: START: telnet pid=1241 from=10.10.10.2
root@BCNMB-A:~#
root@BCNMB-A:~# cat /var/log/syslog | grep "fail"
Jul 31 13:27:54 BCNMB-A dhclient: receive_packet failed on eth0.800: Network is down
Aug 29 12:07:18 BCNMB-A dhclient: receive_packet failed on eth0.800: Network is down
root@BCNMB-A:~#

36
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

SystemEventLog SEL
System Event Log contains information about changes in power feeding,
restarts, external alarms etc.
SEL log can be printed out with ipmitool and mch_cli and filtered normally with
grep.
ipmitool
root@BCNMB-A:~# ipmitool sel
1 | 05/23/2014 | 15:37:15
2 | 05/23/2014 | 15:37:15
3 | 05/23/2014 | 15:37:22
4 | 05/23/2014 | 15:37:23
5 | 05/27/2014 | 14:11:00

list
| Platform
| Platform
| Platform
| Platform
| Platform

mch_cli
root@BCNMB-A:~# mch_cli SEL list
1 | 05/23/2014 | 15:37:15 | BCNMB-A
2 | 05/23/2014 | 15:37:15 | BCNMB-A
3 | 05/23/2014 | 15:37:22 | BCNMB-A
4 | 05/23/2014 | 15:37:23 | BCNMB-A
5 | 05/27/2014 | 14:11:00 | BCNMB-A

Alert
Alert
Alert
Alert
Alert

|
|
|
|
|

Main
Main
Main
Main
Main

#0xfe
#0xfe
#0xfe
#0xfe
#0xfe

|
|
|
|
|

Pair
Pair
Pair
Pair
Pair

Board
Board
Board
Board
Board

|
|
|
|
|

Platform
Platform
Platform
Platform
Platform

root@BCNMB-A:~# mch_cli SEL list | grep "Cold reset"


58 | 06/12/2014 | 14:24:29 | BJC-A | CPU Slot 2 | System
59 | 06/12/2014 | 14:24:29 | BJC-A | CPU Slot 3 | System
5a | 06/12/2014 | 14:24:29 | BJC-A | CPU Slot 1 | System
5d | 06/12/2014 | 14:24:29 | BJC-A | CPU Slot 8 | System

37
1/10/17
Confidential

0
1
1
0
0

alarm
alarm
alarm
alarm
alarm

|
|
|
|
|

Alert
Alert
Alert
Alert
Alert

Event
Event
Event
Event

Nokia 2014 - File Name - Version - Creator - DocID

Deasserted
Deasserted
Asserted
Asserted
Deasserted

#0xfe
#0xfe
#0xfe
#0xfe
#0xfe

#0x81
#0xa1
#0x61
#0x51

"External
"External
"External
"External
"External

"Cold
"Cold
"Cold
"Cold

reset"
reset"
reset"
reset"

alarm"
alarm"
alarm"
alarm"
alarm"
|
|
|
|

|
|
|
|
|

Pair
Pair
Pair
Pair
Pair

0
1
1
0
0

alarm
alarm
alarm
alarm
alarm

|
|
|
|
|

Deasserted
Deasserted
Asserted
Asserted
Deasserted

SystemEventLog SEL
External alarm in SEL
In the example below external alarm switch has been opened for 6sec
root@BCNMB-A:~# mch_cli SEL list
1 | 11/28/2013 | 16:11:50 | Platform Alert #0xfe | Pair 0 alarm | Deasserted
2 | 11/28/2013 | 16:11:56 | Platform Alert #0xfe | Pair 0 alarm | Asserted
root@BCNMB-A:~#

Voltage issues in SEL


root@BCNMB-A:~#
9d2 | 11/28/2013
9d3 | 11/28/2013
9d4 | 11/28/2013
9d5 | 11/28/2013

38
1/10/17
Confidential

mch_cli SEL list | grep "Voltage"


| 10:23:24 | Voltage #0x17
| 10:23:24 | Voltage #0x18
| 10:23:26 | Voltage #0x1a
| 10:23:26 | Voltage #0x1e

|
|
|
|

Lower
Lower
Lower
Lower

Non-recoverable
Non-recoverable
Non-recoverable
Non-recoverable

Nokia 2014 - File Name - Version - Creator - DocID

going
going
going
going

low
low
low
low

|
|
|
|

Reading
Reading
Reading
Reading

1.27 < Threshold 1.00 volts


1.01 < Threshold 0.70 volts
144 < Threshold 44 volts
3.19 < Threshold 3.00 volts

SystemEventLog SEL
Add-in card restarts and FRU state changes are also visible in SEL
root@BCNMB-A:~# mch_cli SEL list | grep "CPU
1 | 06/17/2014 | 13:02:18 | BCNOC-A | CPU
2 | 06/17/2014 | 13:02:33 | BCNOC-A | CPU
3 | 06/17/2014 | 13:03:34 | BCNOC-A | CPU
4 | 06/17/2014 | 13:03:35 | BCNOC-A | CPU
5 | 06/17/2014 | 13:03:37 | BCNOC-A | CPU
6 | 06/17/2014 | 13:03:49 | BCNOC-A | CPU
7 | 06/17/2014 | 13:03:51 | BCNOC-A | CPU
8 | 06/17/2014 | 13:03:52 | BCNOC-A | CPU
9 | 06/17/2014 | 13:03:53 | BCNOC-A | CPU
a | 06/17/2014 | 13:04:05 | BCNOC-A | CPU
root@BCNMB-A:~#

Slot
Slot
Slot
Slot
Slot
Slot
Slot
Slot
Slot
Slot
Slot

8"
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |

System Boot Initiated #0x4d "RESET_TYPE" | Initiated by hard reset |


OS Boot #0x4c "BOOT" | U-boot ready | Asserted
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M5 from M4 cause
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M6 from M5 cause
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M1 from M6 cause
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M2 from M1 cause
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M3 from M2 cause
System Boot Initiated #0x4d "RESET_TYPE" | Initiated by hard reset |
FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M4 from M3 cause
OS Boot #0x4c "BOOT" | U-boot ready | Asserted

SEL records1 & 2 above are written in case in unit is restarted with ZUSU
SEL records 3 to A are related to case where BOC-A add-in card was deactivated and activated from LMP and add-in card FRU state changes.
FRU states:
M0 = FRU not installed (Add-in card status LED Off)
M1 = FRU installed but inactive (Color of status LED: Blue)
M2 = Waiting for activation request (Color of status LED: Blue)
M3 = Activation in progress
M4 = Active (Color of status LED: Red, Green for application usage)
M5 = Waiting for deactivation request (Color of status LED: Red)
M6 = Deactivation in progress (Color of status LED: Red)

39
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Asserted
3 | Asserted
1 | Asserted
0 | Asserted
2 | Asserted
1 | Asserted
Asserted
0 | Asserted

SystemEventLog SEL
More details can be printed with SEL get
root@BCNMB-A:~# mch_cli SEL list | grep "cause 3"
3 | 06/17/2014 | 13:03:34 | BCNOC-A | CPU Slot 8 | FRU Hot Swap #0x38 "Hot Swap AMC 8" | Transition to M5 from M4 cause 3 | Asserted
root@BCNMB-A:~# mch_cli SEL get 0x3
SEL Record ID
: 0003
Record Type
: 02
Timestamp
: 06/17/2014 13:03:34
Generator ID
: 0020
EvM Revision
: 04
Sensor Type
: ATCA Hotswap
Sensor Number
: 38
Event Type
: Sensor-specific Discrete
Event Direction
: Assertion Event
Event Data (RAW)
: 05340C
Event Interpretation Missing
Description
: Transition to M5
Sensor ID
Entity ID
Sensor Type (Analog)
Sensor Reading
States:

: Hot Swap AMC 8(0x38)


: 193.104
: ATCA Hotswap
:

Note that the greps in previous pages are just examples how to read the SEL.
40
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Sensors
BCN and add-in cards have various sensors for Voltage, Temperature watchdog
etc.
Status of these can be investigated via ipmitool and mch_cli

Ipmitool
root@BCNMB-A:~# ipmitool sensor
Sensor Name
| Reading
| Measuring | State | LNR
| LCR
| LNC
| UNC
| UCR
| UNR
-----------------|------------|------------|-------|-----------|-----------|-----------|-----------|-----------|-------PEX Temp
| 48.000
| degrees C | ok
| na
| na
| na
| 105.000
| 110.000
| 115.000
LMP Temp
| 45.000
| degrees C | ok
| na
| na
| na
| 86.000
| 91.000
| 96.000
BCM56512 Temp
| 50.000
| degrees C | ok
| na
| na
| na
| 110.000
| 115.000
| 120.000
BCM56820 Temp
| 71.000
| degrees C | ok
| na
| na
| na
| 105.000
| 110.000
| 115.000

3.3SB
| 3.325
| Volts
| ok
| 3.012
| 3.084
| 3.156
| 3.470
| 3.542
| 3.614
1.1V
| 1.092
| Volts
| ok
| 0.900
| na
| na
| na
| na
| 1.500
1.8V
| 1.828
| Volts
| ok
| 1.500
| na
| na
| na
| na
| 2.004
0.9V
| 0.924
| Volts
| ok
| 0.700
| na
| na
| na
| na
| 1.100
12V
| 12.033
| Volts
| ok
| 11.225
| na
| na
| na
| na
| 12.841

41
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Sensors
Ipmitool gives quick list of sensors and their status.
mch_cli instead can be used for more detailed troubleshooting
mch_cli
ShowSensor gives the list of available sensors,
root@BCNMB-A:~# mch_cli ShowSensor
Entity: CPU 8
...
VCC CPU0
0x01
1.05V
0x01
12V
0x01
1.5V
0x01
1.8V
0x01
5V
0x01
...

Logical Unit Number and sensor number

0x4b
0x50
0x4a
0x49
0x4d
0x44

ReadSensor command give details for the particular sensor


root@BCNMB-A:~# mch_cli ReadSensor 1 0x4a
Sensor: 12V
Lun:
0x01
Number: 0x4a
Value: 11.940000
Syntax: mch_cli ReadSensor <lun> <sensor number>

42
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Debug script

Debug script is used to collect all the BCN logs & majority of
the configurations with one script.

43
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Debug script
BCN debug script in /opt/bcntools can be used to collect BCN level info to one
zip file.
Script collects configuration related to BCN, switches, add-in cards as well the
diagnostic
information
like sensors and SEL
described
in previous
pages.
Script collects
debug information
only from the active
flash bank.
root@BCNMB-A:~#
cd /opt/bcntools/
root@BCNMB-A:/opt/bcntools# ./debug.sh
-> /tmp/debug, Output directory created!!!
-> --------------------------------------------------------------------------------> Start to gather debug information...
-> DATE : Thu Jun 12 16:51:15 UTC 2014
-> --------------------------------------------------------------------------------> Get tar /var/log ...
-> Get VCMC FRU ...
-> END @Thu Jun 12 16:55:54 UTC 2014
->
->
-> All the information are into
/tmp/debug_65_1_EPA2166854_20140612.tgz.
Result is stored into /tmp/ folder with name debug_xyz_date.tgz.
Note that the file name has to be shortened with mv
command before possible FTP upload to OMU file system.

44
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

In the collected data there is eg. Restart log, switch configurations


and System even logs in separate files.

BSAC

Detailed BSAC troubleshooting can be done locally from


BSAC via LMP.

45
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BSAC
BSAC (PTUM) plug-in unit 1/4
Local connection to BSAC is taken with telnet and ssh from LMP to the BSAC
VLAN800 address
Telnet is used to connect synch application and ssh to WindDriver linux in
BSAC.
root@BCNMB-A:~# telnet 10.10.10.51 10000
Trying 10.10.10.51...
Connected to 10.10.10.51.
Escape character is '^]'.
Linux 2.6.27.39-grsec (BCNSA-A) (16:25 on Monday, 23 February
1970)

root@BCNMB-A:~# ssh 10.10.10.51


The authenticity of host '10.10.10.51 (10.10.10.51)' can't be
established.
RSA key fingerprint is
8a:36:a4:e1:ac:75:7b:9e:bc:6e:1e:dd:db:a6:c6:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.10.51' (RSA) to the list of known
hosts.
root@10.10.10.51's password:

Welcome to BCNSA-A AMC Command-Line


Wind River Linux glibc_cgl (cgl) 3.0.2
User: root
Password: ****
(BCNSA-A)>

root@BCNSA-A:/root>
Session is closed with exit command

Session is closed with quit command

46
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

BSAC
BSAC (PTUM) plug-in unit 2/4
BSAC configuration can be checked from application side. Has to be inline with
ZDYB;
(BCNSA-A)>mgmt
ZDYB:;
(BCNSA-A)(mgmt)>sync
(BCNSA-A)(mgmt/sync)>ieee1588v2
(BCNSA-A)(mgmt/sync/1588v2)>view sync
Synchronization Mode:
Protocol Transmit Method:
Master IP Addr:
IEEE1588v2 Two-Step/One-Step Clock:
IEEE1588v2 Two-Way/One-Way:
Unicast Negotiation Feature:
Client IP Addr:
Client GateWay:
Client NetMask:
Client Vlan Option:
PTPv2 Packet Rate:
PTPv2 Packet Size:
PTPv2 UDP Event Port:
PTPv2 General Port:
Framer Operation Mode:
Framer Clock Out Option(MAIN):
Framer Clock Out Option(REDU):
TCLKB Clock Out Option:
Quality Level Option:

47
1/10/17
Confidential

mcBSC
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[

PTP_CLIENT (OC) ]
Unicast ]
10.9.159.19 ]
Two-Step ]
Two-Way ]
Turn On ]
10.255.164.198 ]
10.255.164.193 ]
255.255.255.248 ]
Turn Off ]
64 ]
64 ]
319 ]
320 ]
E1 ]
Turn On ]
Turn On ]
Turn On ]
Turn Off ]

Nokia 2014 - File Name - Version - Creator - DocID

mcKOFF

2014-06-12

INTERROGATE PTUM SYNCHRONIZATION SETTING


LOCATION
--------

OPERATION ROLE
---------------

CAST MODE
---------

STATUS
--------

PTUM-1

IEEE1588-CLIENT

UNICAST

ACTIVE

LOCAL IP ADDRESS : 10.255.164.198/29


REMOTE IP ADDRESS: 10.9.159.19
GATEWAY IP ADDRESS: 10.255.164.193

15:09:52

BSAC
BSAC (PTUM) plug-in unit 3/4
Application log can be checked from /var/log in BSAC WR linux.
root@BCNMB-A:~# ssh 10.10.10.51
root@10.10.10.51's password:
Wind River Linux glibc_cgl (cgl) 3.0.2
root@BCNSA-A:/root> tail -f /var/log/sync_mgmt_app.log | grep "PH"
[6556370s:970505us]ptpSpllEvent: current spll stat is 1 OFFSET: 0 PH:
[6556372s:970598us]ptpSpllEvent: current spll stat is 1 OFFSET: 0 PH:
[6556374s:970687us]ptpSpllEvent: current spll stat is 4 OFFSET: 0 PH:
[6556376s:970776us]ptpSpllEvent: current spll stat is 5 OFFSET: 0 PH:

7.999687512
7.999687512
7.999687512
7.999687512

In the example above only the logwriting that contain phase printout PH are filtered with grep.
Description of the states
State 1: Free run (there is no any sync packages received.)
State 2: holdover(there is no long sync packages received after a long time )
State 3: acquiring(got sync packages, and PTP stack is trying to lock the frequency )
State 4: frequency locked and phase not aligned.
State 5: frequency locked and Phase aligned

48
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

0-I
0-I
0-I
0-I

1-F
1-F
1-F
1-F

2-H
2-H
2-H
2-H

3-A
3-A
3-A
3-A

4-FL
4-FL
4-FL
4-FL

5-FA
5-FA
5-FA
5-FA

BSAC
BSAC (PTUM) plug-in unit 4/4
Telnet session to application can be also taken from WR Linux in BSAC.
IP address to the BSAC is allocated via DHCP and the default addresses are
PTUM0 10.10.10.50
PTUM1 10.10.10.51

49
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

DPDUMP

DPDUMP is used to record trace from BJC-A unit ifeth or vlan


interfaces.

50
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

DPDUMP
With mirroring in extension switch its not possible to monitor mcBSC internal
traffic due the fact that internal VLANs are visible only in main switch.
DPDUMP service terminal extension in BJC-A plug in units can be used for
internal monitoring.
Before DPDUMP logs can be read with wireshark it has to be converted with
Monitoring Vlan20 in OMU with DPDUMP
d2w
software.
ZDDS:OMU;
ZLE:2,DPDUMPGX
Z2T:-xx,-s,0,-n,-i,vlan20
Vlan 20 is used for management, control, dhcp and SW loading for BOC-As

51
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

DPDUMP
Vlan20 monitoring from OMU in ETMA restart 1/2.

DHCP discover from ETMA


Client identifier is used by OMU to detect the
unit
00:ff:ff:ff:ff:08:ff:ff:02:41:0d:21:01:ff:ff:ff
08 = cpu_id (track)
02 = NodeNumber
Racknumber
(65d 2014
in hex)
5241 =1/10/17
Nokia
- File Name - Version - Creator - DocID
Confidential

DHCP Offer from OMU to ETMA


ETMAs IP = 10.0.1.31
SW loading source = 10.0.1.10
Boot file = METASCRA.IMG

DPDUMP
Vlan 20 monitoring from OMU in ETMA restart 2/2.
DHCP behavior can be followed also from the boot log of the add-in card
ETMAs IP = 10.0.1.31
SW loading source = 10.0.1.10
Boot file = METASCRA.IMG

53
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

DPDUMP
SNMP monitoring 1/3
In mcBSC SNMP server is located in MCMU.
SNMP behavior can be investigated by monitoring VLAN800 in MCMU
Monitoring Vlan800 in MCMU with DPDUMP
ZDDS:MCMU,<WO_MCMU>;
ZLE:2,DPDUMPGX
Z2T:-xx,-s,0,-n,-i,vlan800
In DPDUMP port filter can be used also like in SNMP case 162
Z2T:-xx,-s,0,-n,-i,vlan800,port,162
Similarly SNMP & port can be used as a filer in wire shark

54
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

DPDUMP
SNMP monitoring 2/3

In above interface 0/27 was shut down in main switch of


MCBC-0
Switch sends SNMP trap to MCMU
Port indetification 27
Admin status = 2
Oper status = 2
Status 1 = up, 2 = down.

55
1/10/17
Confidential

SNMP trap above causes the alarm 3292 for MCBC-100 interface 1B=27.
<HIST> mcKOFF
** ALARM ........

OMU

TRANSM
2014-05-21
ZYDEMI

(0206) 3292 INTERNAL LAN LINK BROKEN


PTUM 0d BSAC_A 0d 0000 MCBC 100d BCN_A 0d 001B 255d

Nokia 2014 - File Name - Version - Creator - DocID

15:48:29.01

DPDUMP
SNMP monitoring 3/3

In above interface 0/27 was opened in main switch of


MCBC-0
Switch sends SNMP trap to MCMU
Port indetification 27
Admin status = 1
Oper status = 1
Status 1 = up, 2 = down.

56
1/10/17
Confidential

SNMP trap above causes the alarm 3292 cancelation for MCBC-100 interface 1B=27.
<HIST> mcKOFF
.. CANCEL ........

OMU

TRANSM
2014-05-21
ZYDEMI

(0206) 3292 INTERNAL LAN LINK BROKEN


PTUM 0d BSAC_A 0d 0000 MCBC 100d BCN_A 0d 001B 255d

Nokia 2014 - File Name - Version - Creator - DocID

15:49:35.29

TCPDUMP

TCPDUMP is used to record trace from BCN unit ifeth or vlan


interfaces.

57
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

TCPDUMP
SNMP investigation in LMP with tcpdump.

In the example above Vlan800 SNMP port is monitored with tcpdump in LMP
tcpdump -i eth0.800 -s 0 port 162 -w capture.pcap
Interface = vlan800, port = 162, file = capture.pcap
NOTE! It is good idea to log into the file in /tmp folder.

58
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

TCPDUMP
RMCP investigation in LMP with tcpdump.

In the example above Vlan800 RMCP port is monitored with tcpdump in LMP
tcpdump -i eth0.800 -s 0 port 623 -w /tmp/pet_events2.pcap
Interface = vlan800, port = 623, file = /tmp/pet_events2.pcap

59
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

MEGAMON

In mcBSC MEGAMON can be used to collect logs from BJC-A


units similarly as DX units in Flexi BSC

60
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Megamon in mcBSC
mcBSC supports megamon usage via external ports in extension switch.
MCBC-0
Picture on the let illustrates the megamon vlan usage in MCBC-0
BJC-A add-in cards are currently supporting megamon.
BOC-A support is under design and will be released later.
In MCBC-0 main switch megamon vlan is configured to OMU,
MCMU-0, BCXUs 0 & 1, HiGig interfaces and to another
interface towards extension switch.
In MCBC-0 extension switch megamon vlan is configured to
corresponding port towards main switch and to selected probe
port. In this example probe port where megamon is connected is
interface 0/10 (SFP 16)

More details in attached SWTN 905.


Adobe Acrobat
Document
61
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Megamon in mcBSC
Megamon has to be configured to every MCBC module
MCBC-1
Picture on the let illustrates the megamon vlan usage in MCBC-1
In MCBC-1 main switch megamon vlan is configured to MCMU-1,
BCXUs 2 & 3 and HiGig interfaces.
Megamon vlan is configured to extension switch only in the BCN
where megamon is connected which in this example is MCBC-0.
In the rest of the MCBCs megamon is configured similar way.
- EL0 Interfaces to BJC-A cards
- HiGig interfaces

62
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

External traffic

External traffic can be monitored from extension switch of


the BCN unit.

63
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Monitoring external traffic


Monitoring of external traffic can be done in BCN module with mirroring directly
from BCNs extension switch.
Configuration of mirroring in extension switch
(HusaBox1ExtSwitch) #configure
(HusaBox1ExtSwitch) (Config)# monitor session
(HusaBox1ExtSwitch) (Config)# monitor session
(HusaBox1ExtSwitch) (Config)# monitor session
(HusaBox1ExtSwitch) (Config)# monitor session
(HusaBox1ExtSwitch) #show monitor session 1
Session ID
---------1

Admin Mode
---------Enable

Probe Port
---------0/6

1
1
1
1

destination interface 0/6


source interface 0/25
source interface 0/26
mode

Mirrored Port
------------0/25
0/26

Type
----Rx,Tx
Rx,Tx

This setting will mirror all the traffic between main & extension switch to port 6 which is SFP 12.

With the described example all the traffic is mirrored to probe port 0/6 (SFP12).
Port load can be decreased by mirroring only either one of the ports towards
main switch based on the fault under investigation.
64
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Monitoring external traffic


Actual trace is taken with laptop / PC with wireshark from 1G probe port.
Redundant connection to site equipment has to be disabled while trace is
taken.
MCBC-0
MCBC-1

Mirroring
Actual traffic.

65
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Redundant interfaces in MCBC-1 has to be disabled while trace is


taken by shutting down the ports in extension switch or in site
equipment.

Monitoring external traffic


Filtering Pabis Radio signalling from external traffic.
ZOYV:IUA:;
INTERROGATING SCTP ASSOCIATION DATA
SCTP ASSOCIATION ASSOC
NAME
IND
UNIT
---------------- ----- ------BTS1SCTPOM1
0 BCXU-2
BTS1SCTPT01
1 BCXU-2

SCTP
USER
----IUA
IUA

ROLE
-----SERVER
SERVER

PARAMETER SET
NAME
---------------AFAST
AFAST

STATE
------------ASP-ACTIVE
ASP-ACTIVE

ZOYV:IUA:NAME=BTS1SCTPT01:A:;
INTERROGATING SCTP ASSOCIATION DATA
SCTP ASSOCIATION ASSOC
NAME
IND
UNIT
---------------- ----- ------BTS1SCTPT01
1 BCXU-2

SCTP
USER ROLE
----- -----IUA
SERVER

PARAMETER SET
NAME
STATE
---------------- ------------AFAST
ASP-ACTIVE

SOURCE ADDRESS 1 . . . . . . . : 10.255.164.116


SOURCE ADDRESS 2 . . . . . . . :

In the example above filtering is done based on IUA SCTP association destination
address & Radio Signaling Link protocol.
(ip.addr == 10.255.164.180) && (rsl)
Result is radio signaling within one SCTP link.
Details for the signalling can be found from
3GPP TS 48.008, 3GPP TS 48.058, NED DN9814062
66
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

SOURCE PORT . . . . . . . . . : 49153


PRIMARY DEST. ADDRESS . . . . : 10.255.164.180/29
SECONDARY DEST. ADDRESS

. . . :

DESTINATION PORT . . . . . . . : 49153


MAX. DATA STREAM COUNT . . . . : 64
SPECIFICATION VERSION . . . . : 1
COMMAND EXECUTED

Monitoring external traffic


Filtering Sigtran from external traffic 1/2.
OYI:NAME=LIISA:A:;
mcBSC

mcKOFF

2014-06-11

10:12:57

INTERROGATING ASSOCIATION SET DATA


ASSOCIATION SET NAME
-------------------LIISA

ASSOC SET ID
-----------0

ASSOC.
ASSOC ID
IND
UNIT
IN UNIT
--------- --------0
BCXU-2
1

SCTP USER
--------M3UA

ROLE
-------CLIENT

PARAMETER SET
NAME
STATE
---------------- -------------------IETF
ASP-ACTIVE

SOURCE ADDRESS 1 . . . . . : 10.255.164.165


SOURCE ADDRESS 2 . . . . . :
SOURCE PORT . . . . . . . : EPHEMERAL
PRIMARY DEST. ADDRESS . . : 10.8.192.5/24
SECONDARY DEST. ADDRESS

. :

DESTINATION PORT . . . . . : 2905


DATA STREAM COUNT . . . . : 16

In the example above filtering is done based on M3UA SCTP association destination
address
(ip.addr == 10.8.192.5)
Result is signaling within one M3UA SCTP link.
Details for the signalling can be found from
3GPP TS 48.008, 3GPP TS 48.058, NED DN9814062
67
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

SPECIFICATION VERSION .
TRAFFIC MODE . . . . . .
ASP MESSAGES . . . . . .
REGISTRATION REQUEST . .
SSNM MESSAGES BROADCAST
NETWORK APPEARANCE . . .
ASP MESSAGES IN IPSP . .
ROUTING CONTEXT . . . .
FIRST DATA STREAM NUMBER

.
.
.
.
.
.
.
.
.

:
:
:
:
:
:
:
:
:

1.0 (RFC)
LOAD-SHARE
YES
YES
NO
8
NO
--1

Monitoring external traffic


Filtering Sigtran from external traffic 2/2.

User can add local and destination and source references as colums to help in the
analysis.

68
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Monitoring external traffic


Filtering Gb from external traffic 1/2.
FXI:NSEI=401:;
mcBSC

mcKOFF

2014-06-11

NETWORK SERVICE VIRTUAL LINK PARAMETERS:


PSE-00 NSE-00401
================
LOCAL UDP PORT: 50303
NSE STATUS:
TRAFFIC ALLOWED
NSE PFC MODE:
ON
NSE NRI LIST (BINARY/DECIMAL):
0001000011/67
NSE NRI MASK: 1111111111
PRE CONFIGURED NS-VL:
REMOTE
ID
NAME
IP ADDRESS
----- ---------- --------------00401 NETTA1
10.8.37.37

REMOTE
UDP PORT
-------49999

REMOTE HOST NAME:

In the example above filtering is done based on NS-VL remote IP address


(ip.addr == 10.8.37.37)
Result is signaling of one NS-VL. If there is more dynamic NS-VLs those need to be
added to the filter. Alternatively whole Gb subnet can be filtered with ip.addr ==
10.255.164.0/26
Note! By default Wireshark does not detect NS-LAYER. User needs to define the
decoding for one of the UDP packets between PCUM and SGSN as described above.
69
1/10/17
Nokia 2014 - File Name - Version - Creator - DocID
Confidential

DYNAMIC CONFIGURED NS-VL(S):


ID
----00402

OP
REMOTE
STATE RDW RSW IP ADDRESS
----- --- --- --------------WO-EX 001 001 10.8.37.37

REMOTE
UDP PORT
-------55501

10:49:17

Monitoring external traffic


Filtering Gb from external traffic 2/2.

After proper decoding definition Wirseshark is able to decode the NS-LAYER and
protocols on top of it. Note that if ciphering is used the ciphered LLC frames are not
decoded.

70
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Monitoring external traffic


Filtering ETPSIG-M and ETPSIG-C towards MCTC.
D2I:::MCTC,;
mcBSC

mcKOFF

2014-06-11

12:23:20

INTERROGATE IUA MESSAGING CONFIGURATION


UNIT
--------MCTC-0
BCXU-2
MCTC-0
MCTC-0
BCXU-2
MCTC-0

CONN-TYPE
--------NORMAL
IP ADDR:
IP ADDR:
NORMAL
IP ADDR:
IP ADDR:

SIG-TYPE GW UNIT
-------- -------ETPSIG-M BCXU-2
10.255.164.140
10.255.166.20
ETPSIG-C BCXU-2
10.255.164.132
10.255.166.28

D-CHN
----79
80

COMMAND EXECUTED
IUA MESSAGE INTERFACE OVER UDP CONFIGURATION HANDLING COMMAND <D2_>
<

Before wireshark is able to decode ETPSIG the dx_mcETP.lua and


TCU.lua files have to be copied into wireshark folder. After this
following files have to be added to the end of the init.lua file.
dofile("dx_mcETP.lua")
dofile("TCU.lua")

71
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

dx_mcETP.lua

TCU.lua

Traffic flow

Following pages give the details how to troubleshoot the


traffic flow in mcBSC units.

72
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


Ongoing CS, PS and CS data calls and related configurations can be
interrogated from ETM and PCUM terminals.
For the CS call resources are allocated from ETME and ETMA
For the PS call resources are allocated from ETME and PCUM
For the CS Data call from ETME. In practice DL flow from mcTC goes through
ETMA but ETMA acts only as a router for the IP packets.
Note that the examples in following pages are from S16.1 MP0.0.2

73
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


CS call
Command detpcs gives detailed information of ongoing calls in ETME and
ETMA
ETME-A/1>detpcs
ETMA-A/0>detpcs
Num CS calls Active :: 2
CCH_STATE :: 1, CCH_STATE_NORMAL

Total Running Calls = 1


ETMA-A/0>detpcs all

ETME-A/1>detpcs all
Call 1 Info:
---------------------------------------------------ETP BCF ID/RCP
:: 0 / 16392
BCF U-plane IP
:: 0xaffa5b4
BCSU Index/Proc Id
:: 0x4f74 / 0xaf03
ETPA TLA
:: 0xa00030a / 0xffd2
ETPT/E MUX Port
:: 49152

BASE UDP PORT


= 49152
Call 1 Info:
---------------------------------------------------ConnectionID/LocalID :: 0x1a000007/20
ETPT/E TLA
:: 10.255.165.180/0xc000
MGW TLA
:: 10.8.3.86/0x2c0e
BCSU Index/Proc ID
:: 0x4f75/0x797
CallState
:: ACTIVE
Codec Type
:: CODEC_AMR_NB
ETP_CALL_ID
:: 0/4
Total Running Calls = 1
Total No of HO Calls Running = 0
Total No of LS with LI Calls = 0
Total No of LS without LI Calls = 0
Mca_feature_status = Disable

Call 2 Info:
---------------------------------------------------ETP BCF ID/RCP
:: 0 / 16400
BCF U-plane IP
:: 0xaffa5b4
BCSU Index/Proc Id
:: 0x4f74 / 0xaf05
ETPA TLA
:: 0xa00030b / 0xffc4
ETPT/E MUX Port
:: 49152
Total running calls :: 2

74
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ETME IPs
BCF U-Plane:
ETPA TLA:
ETMA)

affa5b4 = 10.255.165.180
0xa00030b = 10.0.3.11 (EEP address of the

Traffic flow in mcBSC


CS call
duddl and dudul can be used to investigate that the UDP dataflow is
ongoing.
Ongoing calls in ETME
DL UDP-flow
ETME-A/1>duddl
UDP Packets passed to Apps for DL cores are: 9982453
UDP Packets received from Apps for DL cores are: 9787041

Ongoing calls in ETMA


DL UDP-flow
ETMA-A/1>duddl
UDP Packets passed to Apps for DL cores are: 8321425
UDP Packets received from Apps for DL cores are: 8320331

ETME-A/0>duddl
UDP Packets passed to Apps for DL cores are: 9982667
UDP Packets received from Apps for DL cores are: 9787148

ETME-A/0>duddl
UDP Packets passed to Apps for DL cores are: 8324111
UDP Packets received from Apps for DL cores are: 8322721

UL UDP-flow
ETME-A/1>dudul
UDP Packets passed to Apps for UL cores are: 6133179
UDP Packets received from Apps for UL cores are: 10927684

UL UDP-flow
ETMA-A/1>dudul
UDP Packets passed to Apps for UL cores are: 8356575
UDP Packets received from Apps for UL cores are: 8338270

ETME-A/0>dudul
UDP Packets passed to Apps for UL cores are: 6133299
UDP Packets received from Apps for UL cores are: 10927922

ETMA-A/0>dudul
UDP Packets passed to Apps for UL cores are: 8358452
UDP Packets received from Apps for UL cores are: 8344243

Commands need to be given several times in sequence to check


that the values are increasing.

Commands need to be given several times in sequence to check


that the values are increasing.

75
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


PS call (PCU troubleshooting covered in separate training)
PCUM has several service terminal commands for PS troubleshooting.
Most important of these are configuration related commands eg. etp_globals,
dgb and dsegrnw.
ETME configuration in PCUM
PCUM-A/1/10.0.1.21>etp_globals
ETP Configuration Globals:
*************************PCU2-E********************************
DSP state: 3
etp_eth_connect_status:1
etp_link_status_alarm_2e:0
eth_link_state:1
ip_ver:0
etp_ip_addr : 10.0.3.129
ETP IP has to be the PEP IP of the defined ETME which can be
checked with ZESL::ETME,1:PCU:;
PEP link state:
PCUM-A/3/10.0.1.21>dals
---------------------------------------------------------------> Command dals executed on [25/6/2014 11:36:8:313093] <--------------------------------------------------------------ABIS LINK STATE: LINK WORKING
PCUM-A/3/10.0.1.21>

Gb interface in PCUM
PCUM-A/1/10.0.1.21>dgb all
************************ Start of Trace ************************
Bearer Network in use is IP
Number of NSEI configured::1
BVCs in Use :
Segment ID::1
BVC ID::10001 POOL ID::NONE
Number of BVCs Configured: 1
PFC Feature Mode[0]: IMSGW_TRUE
NSEI: 118
MOCN Support: NO
NSVCs information :
Local Endpoint: IP Address::10.255.165.5
UDP Port::51118
NSVC ID::118
Remote Endpoint: IP Address::10.8.37.37
UDP Port::51118
Remote Data Weight::1
Remote Signalling Weight::1

Configuration in PCUM has to be inline with ZFXI and ZFXL.

76
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


PS call (PCU troubleshooting covered in separate training)
Current traffic in PCUM
PCUM-A/1/10.0.1.21>dpcutbf
-----------------------------------------------------------Command dpcutbf executed on [23/6/2014 15:4:39:143822] <--------------------------------------------------------------***********************START OF TRACE****************************
SEG-ID Num-TRX DSP-ID
Num-UL-TBFs
Num-DL-TBFs
231
1
0
3
3
----------------------------------------------------------------TOTAL:
3
3
----------------------------------------------------------------************************END OF TRACE****************************

77
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

PCUM resources reserved based on IMSI


PCUM-A/1/10.0.1.21>ddimsi imsi 520181021512110
-----------------------------------------------------------Command ddimsi executed on [23/6/2014 15:7:59:673804]
-----------------------------------------------------------Dedicated IMSI Entry Table :
-----------------------------------------------ENCODED IMSI ID :
-------- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -------------00000000 59 02 81 01 12 15 12 01
Y.......
-------- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ------------------------------------------------------------IMSI ID: 520181021512110
-----------------------------------------------Dedicated IMSI Ctx Ptr [0x5d003838] :
Dedicated TLLI Ctx Ptr [0x586548c8] :
DTM MS Ctx Ptr [0x78e551f0] :
-----------------------------------------------Dedicated IMSI Context Fields Table Starts :
-----------------------------------------------ABIPRB address : [0x4377]
Alloc Req Rxd : [NO]
ReAlloc Req Rxd : [NO]

Traffic flow in mcBSC


CS-data call
Command detpcs gives detailed information in ETME and ETMA
ETME-A/1>detpcs

ETMA-A/0>detpcs

Num CS calls Active :: 2


CCH_STATE :: 1, CCH_STATE_NORMAL

Total Running Calls = 0


ETMA does not have resources reserved for the CS-data call.

ETME-A/1>detpcs all
Call 1 Info:
---------------------------------------------------ETP BCF ID/RCP
:: 0 / 16408
BCF U-plane IP
:: 0xaffa5b4
BCSU Index/Proc Id
:: 0x4f74 / 0xaf07
ETPA TLA
:: 0xaffa60c / 0xc0a8
ETPT/E MUX Port
:: 49152
Call 2 Info:
---------------------------------------------------ETP BCF ID/RCP
:: 0 / 16416
BCF U-plane IP
:: 0xaffa5b4
BCSU Index/Proc Id
:: 0x4f74 / 0xaf09
ETPA TLA
:: 0xaffa60c / 0xc0aa
ETPT/E MUX Port
:: 49152
Total running calls :: 2

78
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ETME IPs
BCF U-Plane:
ETPA TLA:

affa5b4 = 10.255.165.180
affa60c = 10.255.166.12 (ATER IF IP in mcTC)

ETME sends the UL packets to mcTC directly. DL packets are


routed via ETMA.

Traffic flow in mcBSC


Routes
For all the units it is essential that the routes are in order to secure proper
traffic
flow.
OMU routes
OMU needs physical default route to
ZQRS:OMU::ROU:;

gateway.

Routing tables
Internet:
Destination
default
10.9.192.240/29
10.9.192.241
10.10.10/24
10.10.10.10
10.10.10.11
10.200.206/23
10.200.208/23
127.0.0.1

Gateway
10.9.192.241
link#5
00:00:0c:9f:f0:5a
link#4
00:d0:c9:c9:4f:f3
00:d0:c9:c9:94:02
link#2
link#3
127.0.0.1

Flags
UG
UC
UHLW
UC
UHLW
UHLW
UC
UC
UH

Refs
0
0
2
0
1
1
0
0
0

Use
33601
0
0
0
385052
402820
0
0
44

Mtu
Netif Expire
1500 vlan3810
1500 vlan3810
1500 vlan3810
1130
1500 vlan800
1500 vlan800
1189
1500 vlan800
1166
1500
el0
1500
el1
16384
lo0

INTERROGATED STATIC ROUTES


GATEWAY ADDRESS
UNIT
->DESTINATION
---------------------- -------------------------------------------OMU
10.9.192.241
->DEFAULT ROUTE

79
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ROUTE
TYPE NBR
----- ----PHY
1

the O&M

Traffic flow in mcBSC


Routes
MCMU routes
ZQRS:MCMU,0::ROU:;
mcBSC

mcAmaretto

MCMU needs logical IP address based route to


the Sigtran gateway.
2014-06-25

Local IP address based default gw:


source address
default gw
10.255.165.164
10.255.165.161

12:01:53

interface
vlan3832

ZQKO:MCMU,1:;
INTERROGATED LOCAL IP ADDRESS BASED DEFAULT GATEWAY ROUTES
ROUTE
UNIT
TYPE NBR
--------------------- ----- ---------MCMU-1
LOG
(1)
SOURCE ADDRESS: 10.255.165.164
DEFAULT GW
: 10.255.165.161

80
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


Routes
BCXU routes
ZQRS:BCXU,0::ROU:;
mcBSC

mcAmaretto

2014-06-25

12:05:32

UNIT: BCXU-0
Routing tables
Local IP address based default gw:
source address
default gw
10.255.165.165
10.255.165.161
10.255.165.132
10.255.165.129
10.255.165.116
10.255.165.113

interface
vlan3832
vlan3836
vlan3844

ZQKO:BCXU,0:;
INTERROGATED LOCAL IP ADDRESS BASED DEFAULT
GATEWAY ROUTES
ROUTE
UNIT
TYPE NBR
--------------------- ----- ---------BCXU-0
LOG
2
SOURCE ADDRESS: 10.255.165.165
DEFAULT GW

: 10.255.165.161

BCXU-0
LOG
4
SOURCE ADDRESS: 10.255.165.116
DEFAULT GW

: 10.255.165.113

BCXU-0
LOG
6
BCXU needs logical IP address based route to
SOURCE ADDRESS: 10.255.165.132
Sigtran (vlan3832),
ETPSIG-M (this mcBSC is client),
DEFAULT GW
: 10.255.165.129
ETPSIG-C (vlan3836) and
OMU & TRXSIG (vlan3844) gateways.
EPTSIG-M & C for mcTC and ETPSIG-M only in
master mcBSC.
81
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Traffic flow in mcBSC


Routes
ETME routes
ETME-A/1>route
Kernel IP routing table
Destination
Gateway
10.255.166.8
10.255.165.81
10.255.165.80
*
10.255.165.64
*
10.0.1.0
*
10.0.3.128
*
10.0.3.0
*
default
10.255.165.65

ETME needs logical routes to the Pater (47)


and Pabis (48)gateways.
Genmask
255.255.255.248
255.255.255.240
255.255.255.240
255.255.255.128
255.255.255.128
255.255.255.128
0.0.0.0

Flags
UG
U
U
U
U
U
UG

Metric
0
0
0
0
0
0
0

Ref
0
0
0
0
0
0
0

Use
0
0
0
0
0
0
0

Iface
IFETH1.3847
IFETH1.3847
IFETH1.3848
IFETH0
IFETH0.24
IFETH0.23
IFETH1.3848

ZQKB:ETME,1:
INTERROGATED STATIC ROUTES
GATEWAY ADDRESS
UNIT
->DESTINATION
---------------------- -------------------------------------------ETME-1
10.255.165.65
->DEFAULT ROUTE
ETME-1
10.255.165.81
->10.255.166.8/29
COMMAND EXECUTED

82
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ROUTE
TYPE NBR
----- ----LOG
4
LOG

11

Traffic flow in mcBSC


Routes
ETMA routes
ETMA-A/0>route
Kernel IP routing table
Destination
Gateway
10.255.165.180 10.0.3.1
10.255.166.8
10.255.165.81
10.255.165.80
*
10.255.165.96
*
10.0.1.0
*
10.0.3.0
*
default
10.255.165.97

ETMA needs physical routes to the Pater (47)


and AoIP (46) gateways.
Genmask
255.255.255.255
255.255.255.248
255.255.255.240
255.255.255.240
255.255.255.128
255.255.255.128
0.0.0.0

Flags
UGH
UG
U
U
U
U
UG

Metric
0
0
0
0
0
0
0

Ref
0
0
0
0
0
0
0

Use
0
0
0
0
0
0
0

Iface
IFETH0.23
IFETH1.3847
IFETH1.3847
IFETH1.3846
IFETH0
IFETH0.23
IFETH1.3846

ZQKB:ETMA,1:
INTERROGATED STATIC ROUTES
GATEWAY ADDRESS
UNIT
->DESTINATION
---------------------- -------------------------------------------ETMA-1
10.255.165.97
->DEFAULT ROUTE
ETMA-1
10.0.3.1
->10.255.165.180/32
ETMA-1
10.255.165.81
->10.255.166.8/29

83
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ROUTE
TYPE NBR
----- ----PHY
3
PHY

PHY

10

Specialty in ETMA is that it needs route to


the BTS subnet via ETMEs EEP address (23).
If this is not defined or working the user
payload is not working.

Traffic flow in mcBSC


Routes
PCUM routes
PCUM-A/0/10.0.1.20>route
Kernel IP routing table
Destination
Gateway
10.255.165.0
*
10.0.1.0
*
10.0.3.128
*
default
10.255.165.1

PCUM needs logical routes to the Gb over IP


(49) gateway.
Genmask
255.255.255.192
255.255.255.128
255.255.255.128
0.0.0.0

Flags
U
U
U
UG

Metric
0
0
0
0

Ref
0
0
0
0

Use
0
0
0
0

Iface
IFETH1.3849
IFETH0
IFETH0.24
IFETH1.3849

ZQKB:PCUM,1:;
INTERROGATED STATIC ROUTES
GATEWAY ADDRESS
UNIT
->DESTINATION
---------------------- -------------------------------------------PCUM-1
10.255.165.1
->DEFAULT ROUTE

84
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

ROUTE
TYPE NBR
----- ----LOG
6

mcTC troubleshooting

Following pages give the details for the mcTC


troubleshooting.

85
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
Rack & Node number are configured similar way as in mcBSC and these both
need to have value 1.
root@BCNMB-A:~# mch_cli GetNodeNumber
1
root@BCNMB-A:~# mch_cli GetRackNumber
1
root@BCNMB-A:~#

VLAN800 address for the BCN in mcTC is allocated via DHCP


root@BCNMB-A:~# cat /etc/sysconfig/network-scripts/ifcfg-eth0.800
DEVICE=eth0.800
VLAN=yes
BOOTPROTO=dhcp
#ONBOOT=no
ONBOOT=yes
PERSISTENT_DHCLIENT=Y
DHCLIENTARGS="-nw"
root@BCNMB-A:~#

86
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
VLAN800 addressing in mcTC is done by MCU.
Renewal of the VLAN800 address is performed frequently which can be
checked from the syslog.
LMP syslog
root@BCNMB-A:~#
Jun 4 21:26:16
Jun 4 21:26:16
Jun 4 21:26:16
Jun 4 21:55:38
Jun 4 21:55:38
Jun 4 21:55:38

87
1/10/17
Confidential

cat /var/log/syslog | grep dhc


BCNMB-A dhclient: DHCPREQUEST on eth0.800 to
BCNMB-A dhclient: DHCPACK from 169.254.128.4
BCNMB-A dhclient: bound to 169.254.128.13 -BCNMB-A dhclient: DHCPREQUEST on eth0.800 to
BCNMB-A dhclient: DHCPACK from 169.254.128.4
BCNMB-A dhclient: bound to 169.254.128.13 --

Nokia 2014 - File Name - Version - Creator - DocID

169.254.128.4 port 67
renewal in 1762 seconds.
169.254.128.4 port 67
renewal in 1421 seconds.

mcTC Troubleshooting
mcTC configuration
Since the MCU is running on Linux as well separate syslog exists in the unit.
MCU syslog
root@BCNMB-A:~# minicom cpu1
[root@MCU-0(scrooge) /root]
# cat /var/log/syslog | grep -i -E 'Jun
Jun 4 21:26:16.026949 info MCU-0 dhcpd:
Jun 4 21:26:16.027040 info MCU-0 dhcpd:
Jun 4 21:28:54.215960 info MCU-0 dhcpd:
Jun 4 21:28:55.004155 info MCU-0 dhcpd:
Jun 4 21:28:55.030116 info MCU-0 dhcpd:
Jun 4 21:28:55.030209 info MCU-0 dhcpd:
Jun 4 21:54:48.058996 info MCU-0 dhcpd:
Jun 4 21:54:48.059093 info MCU-0 dhcpd:
Jun 4 21:55:38.027866 info MCU-0 dhcpd:
Jun 4 21:55:38.027969 info MCU-0 dhcpd:

4.*dhc
DHCPREQUEST for 169.254.128.13 from 00:d0:c9:bb:b6:5d (BCNMB-A) via xaui0
DHCPACK on 169.254.128.13 to 00:d0:c9:bb:b6:5d (BCNMB-A) via xaui0
DHCPDISCOVER from 00:40:43:60:99:b1 via xaui0
DHCPOFFER on 169.254.128.12 to 00:40:43:60:99:b1 via xaui0
DHCPREQUEST for 169.254.128.12 (169.254.128.4) from 00:40:43:60:99:b1 via xaui0
DHCPACK on 169.254.128.12 to 00:40:43:60:99:b1 via xaui0
DHCPREQUEST for 169.254.128.12 from 00:40:43:60:99:b1 via xaui0
DHCPACK on 169.254.128.12 to 00:40:43:60:99:b1 via xaui0
DHCPREQUEST for 169.254.128.13 from 00:d0:c9:bb:b6:5d (BCNMB-A) via xaui0
DHCPACK on 169.254.128.13 to 00:d0:c9:bb:b6:5d (BCNMB-A) via xaui0

169.254.128.13 address is for BCN VLAN800 and 169.254.128.12 is for BSAC

88
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
In mcTC the HW created during commissioning.
HW config can be interrogated in MCU from fsclish shell
Brief list:
[root@MCU-0(scrooge) /root]
# fsclish
root@MCU-0 [scrooge] >
root@MCU-0 [scrooge] > show hardware inventory list brief
/chassis-1/AMC-1 [HDSAM-A]
/chassis-1/AMC-2 [BCNSA-A]
/chassis-1/FT-1 [BMFU-A]
/chassis-1/FT-2 [BMFU-A]
/chassis-1/FT-3 [BAFU-A]
/chassis-1/motherboard-1 [BCNMB-A]
/chassis-1/power-supply-2 [BCNAP-B]
/chassis-1/slot-1 [BCNOC-A]
/chassis-1/slot-2 [BCDSP-A]
/chassis-1/slot-3 [BCDSP-A]
/chassis-1/slot-8 [BCNOC-A]

89
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Detailed info:
[root@MCU-0(scrooge) /root]
# fsclish
root@MCU-0 [scrooge] >
root@MCU-0 [scrooge] > show hardware inventory list detailed
/chassis-1/AMC-1 [HDSAM-A]
Product Manufacturer
:
Product Name
:
Product Part Number
:
Product Version
:
Product Serial Number
:
Product Asset Tag
:
Product File Id
:
Board
Mfg Datetime
:
Board
Manufacturer
:
Board
Name
:
Board
Serial Number
:
Board
Part Number
:
Board
File Id
:

Advantech Co.
MIC-5401
MIC-5401-0000E
A1.02
AKO0773745
hdsam-a_ad_frud_01_10_0000.bin
2011.07.20 11:30:00
ADVANTEC
HDSAM-A
AT112801085
C110598.B3A

mcTC Troubleshooting
mcTC configuration
Current status/availability of the HW can be interrogated in MCU from fsclish
shell
[root@MCU-0(scrooge) /root]
# fsclish
root@MCU-0 [scrooge] > show hardware state list
cabinet-1
chassis-1
LMP-1-1-1
MCU-0
TCU-0
TCU-1
TCU-2
TCU-3
TCU-4
TCU-5
ETP-0
SE-0
PTU-0

90
1/10/17
Confidential

:
:
:
:
:
:
:
:
:
:
:
:
:

unit
unit
node
node
node
node
node
node
node
node
node
node
node

unknown
unknown
available
available
available
available
unplugged
unplugged
unplugged
unplugged
available
available
available

/cabinet-1
/cabinet-1/chassis-1
/cabinet-1/chassis-1/piu-1
/cabinet-1/chassis-1/piu-1/addin-1/CPU-1/core-0,1,10,11,2,3,4,5,6,7,8,9
/cabinet-1/chassis-1/piu-1/addin-2/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-3/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-4/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-5/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-6/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-7/LMP-0/CPU-0/core-0
/cabinet-1/chassis-1/piu-1/addin-8/CPU-1/core-0
/cabinet-1/chassis-1/piu-1/addin-8/CPU-1/core-1,10,11,2,3,4,5,6,7,8,9
/cabinet-1/chassis-1/piu-1/AMC-2/CPU-0/core-0,1

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
Synchronization config in MCU
root@MCU-0 [scrooge] > show hardware clock-sync config all
VCMC configuration : chassis "chassis-1" in cabinet "cabinet-1":
LDAP config:
External interface configuration:
sync-1 : iftype = 2.048MHz, output source = ext-ref
sync-2 : iftype = 2.048MHz, output source = ext-ref
Output source configuration:
sync-1 : enabled
sync-2 : enabled
internal : enabled
PLL input configuration:
Priority:
sync-1 : high
sync-2 : medium
internal : low

General:
Internal:
resource : AMC-2
PTU configuration : chassis "chassis-1" in cabinet "cabinet-1":
Operational mode
: ieee1588v2-slave
TCLKB output
: enabled
Framer mode
: e1
Stack mode
: two-step
Delivery mode
: unicast
Master IP
: 10.9.159.19
Clock Master GW IP
: 10.255.164.193/29
PTPv2 packet rate
: 64
PTPv2 Client configuration:
Client IP
: 10.255.164.197
Client VLAN Status
: disabled

Mode:
sync-1 : Frequency = 2.048MHz
sync-2 : Frequency = 2.048MHz
internal : Frequency = 19.44MHz

BSAC-A VLAN 800 address in mcTC is 169.254.128.12


91
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
ETP configuration file contains the integration related information.
File is located in /tftpboot from where it is loaded to the ETPc during the
restart.
ETPC restart:
[root@MCU-0(scrooge) /root]
# ls -la /tftpboot/ETPCFGDT.cfg
-rw-r--r-- 1 root root 1489 Nov 22 2013 /tftpboot/ETPCFGDT.cfg
[root@MCU-0(scrooge) /root]
# cat /tftpboot/ETPCFGDT.cfg
[ETPCONF]
VLAN_ID_TCUSIG=823
ETP_TCUSIG_IF_IPADD=192.168.51.51
ETP_TCUSIG_IF_MASK=255.255.255.0
VLAN_ID_AOIP=3648
ETP_A_IF_IPADD=10.255.166.4
ETP_A_IF_MASK=255.255.255.248
VLAN_ID_MBSC=3646
ETP_ATER_IF_OAM_IPADD=10.255.166.20
ETP_ATER_IF_OAM_MASK=255.255.255.248

92
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

IFETH0: 10000 Mbps Full duplex, port 0, queue 0


IFETH1: 10000 Mbps Full duplex, port 16, queue 64
/etc/ssh/sshd_config line 77: Unsupported option
GSSAPIAuthentication
/etc/ssh/sshd_config line 79: Unsupported option
GSSAPICleanupCredentials
Starting TFTP to get METCMGTA.elf
TFTP completed
The size of METCMGTA.elf is <5425820> bytes
Starting TFTP to get ETPCFGDT.cfg
TFTP completed
Bringing down the interfaces initially
Making MTD nodes
Installing Driver
Done installation

mcTC Troubleshooting
mcTC configuration
ETP configuration shall be modified only via fsclish in MCU (no direct
modification of ETPCFGDT file)
Refer to the Managing the BSS Transmission document DN98785698
ETP configuration example:
Configure master mcBSC
root@MCU-0 [scrooge] > set etpcfg mbscconf mbsc-ipadd 10.255.164.140 mbsc-mask 255.255.255.248
//ETPSIG-M address in BCXU
Configure TCSU
root@MCU-0 [scrooge] > add etpcfg tcsu tcsun 0 vlantagn 0 cbsc-ipadd 10.255.164.132 cbsc-mask 255.255.255.248
//vlantagn == vlan tag number of the etpsig-c vlan
//cbsc-ipadd == Client BSC IP address (ETPSIG-C address in BCXU)
//cbsc-mask == Client BSC IP mask
Save the changed ETP configuration
root@MCU-0 [scrooge] > commit etpcfg upgrade
Checking IP addresses...
...........................
...Done!
cp -f /tftpboot/tmp/ETPCFGDT.cfg /tftpboot/ETPCFGDT.cfg
cp -f /tftpboot/tmp/ETPCFGDT.cfg /mnt/images/R_MCTC_1.0.WR.20120301B9.6.2_WR/mi
cp -f /tftpboot/tmp/ETPCFGDT.cfg /mnt/config/R_MCTC_1.0.WR.20120301B9.6.2_WR/IN
root@MCU-0 [scrooge] >

93
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC configuration
ETP config interrogation.
ETP configuration shall be inline between MCU and ETPC
ETP configuration in MCU:
root@MCU-0 [scrooge] > show etpcfg tcsu tcsun 0
CBSC_IPADD=10.255.164.132
CBSC_MASK=255.255.255.248
VLAN_CONF_REF=VLAN0CONF
root@MCU-0 [scrooge] > show etpcfg tcsu tcsun 1
CBSC_IPADD=10.255.165.132
CBSC_MASK=255.255.255.248
VLAN_CONF_REF=VLAN0CONF
root@MCU-0 [scrooge] >
Available configs:
etpconf
- Show
gateway
- Show
mbscconf
- Show
mcuconf
- Show
tcsu
- Show
vlan
- Show
vlanintconf - Show

94
1/10/17
Confidential

ETP config in ETP config file


gateway in ETP config file
MBSC config in ETP config file
MCU config in ETP config file
tcsu in ETP config file
VLAN in ETP config file
VLANINT config in ETP config file

Nokia 2014 - File Name - Version - Creator - DocID

ETP configuration in ETPC:


MCETPC/0>cat ETPCFGDT.cfg
[ETPCONF]
VLAN_ID_TCUSIG=823
ETP_TCUSIG_IF_IPADD=192.168.51.51
ETP_TCUSIG_IF_MASK=255.255.255.0
VLAN_ID_AOIP=3648
ETP_A_IF_IPADD=10.255.166.4
ETP_A_IF_MASK=255.255.255.248
VLAN_ID_MBSC=3646
ETP_ATER_IF_OAM_IPADD=10.255.166.20
ETP_ATER_IF_OAM_MASK=255.255.255.248
...
[TCSU0CONF]
CBSC_IPADD=10.255.164.132
CBSC_MASK=255.255.255.248
VLAN_CONF_REF=VLAN0CONF
[TCSU1CONF]
CBSC_IPADD=10.255.165.132
CBSC_MASK=255.255.255.248
VLAN_CONF_REF=VLAN0CONF

mcTC Troubleshooting
mcTC configuration
ETPSIG-M and ETPSIG-C config in ETPC
ETPSIG-M should be configured only to the master mcBSC
ETPSIG-C should be created for all the TCSUs in client mcBSCs
ETPSIG-M
MCETPC/0>etpsig_m_info
---------------------------------------------------------------> Command etpsig_m_info executed on [17/6/2014
22:11:50:801407] <--------------------------------------------------------------********************************************************
The ETPSIG-M information is as follows:
********************************************************
ETP IP address:
10.255.166.20
ETP Port Number:
9900
GW BCSU IP address:
10.255.164.140
GW BCSU Port Number:
9900

ETPSIG-C
MCETPC/0>etpsig_c_info
---------------------------------------------------------------> Command etpsig_c_info executed on [17/6/2014
22:11:47:73497] <--------------------------------------------------------------********************************************************
The ETPSIG-C information is as follows:
********************************************************
TCSU Number
:
0
BCSU IP address :
10.255.164.132
BCSU Port Number :
9900
ETP IP address
:
10.255.166.28
ETP Port Number :
9900
TCSU Number
BCSU IP address
BCSU Port Number
ETP IP address
ETP Port Number

95
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

:
:
:
:
:

1
10.255.165.132
9900
10.255.166.28
9900

mcTC Troubleshooting
mcTC configuration
TCSU configuration in mcBSC. ETPSIG-M to master mcBSC, ETPSIG-C and TCSU
in client mcBSC
TCSU configuration in first client mcBSC
ZWGB::;

ETPSIG-M in mcBSC
ZD2I:M::MCTC,;
mcBSC

mcKOFF

2014-06-19

11:45:59

INTERROGATE IUA MESSAGING CONFIGURATION


UNIT
--------MCTC-0
BCXU-2
MCTC-0

CONN-TYPE SIG-TYPE GW UNIT


--------- -------- -------NORMAL
ETPSIG-M BCXU-2
IP ADDR: 10.255.164.140
IP ADDR: 10.255.166.20

D-CHN
----79

mcKOFF

2014-06-19

11:46:15

INTERROGATE IUA MESSAGING CONFIGURATION


UNIT
--------MCTC-0
BCXU-2
MCTC-0
96
1/10/17
Confidential

CONN-TYPE SIG-TYPE GW UNIT


--------- -------- -------NORMAL
ETPSIG-C BCXU-2
IP ADDR: 10.255.164.132
IP ADDR: 10.255.166.28

D-CHN
----80

Nokia 2014 - File Name - Version - Creator - DocID

TCSU_NUM
-------0

ETPSIGC-DNBR
-----------80

STATE
----WRK

TCSU configuration in second client mcBSC


ZWGB;
TCSU_ID
------1

ETPSIG-C in client mcBSC


ZD2I:C,::MCTC,;
mcBSC

TCSU_ID
------0

TCSU_NUM
-------1

ETPSIGC-DNBR
-----------79

STATE
----WRK

mcTC Troubleshooting
mcTC functionality
mcTC & TCSU functional states can be interrogated with mctc CLI commands
root@MCU-0 [scrooge]
mcTC State=WO-EX
root@MCU-0 [scrooge]
TCSU 0 state=WO-EX
root@MCU-0 [scrooge]
TCSU 1 state=WO-EX
root@MCU-0 [scrooge]

> mctccli getmctcstate


> mctccli gettcsustate tcsu 0
> mctccli gettcsustate tcsu 1
>

SW version informations can be interrogated via CLI as well


root@MCU-0 [scrooge] > mctccli getmctcswver
MCTCMCU
10.1.11
METCKRNA.elf
1.48-0
METCMGTA.elf
1.61-1
METCUPCA.elf
1.61-1
mctdsp.out
1.7
root@MCU-0 [scrooge] > mctccli getmcuswver
mcTC Application Library: SS_mcTCMCU-10.1.11.3804-R_MCTC_1.0.WR.mips.337823.1803
Module Manager: SS_mcTCMCU-10.1.11.3804-R_MCTC_1.0.WR.mips.337823.180913
Transcoder Manager: SS_mcTCMCU-10.1.11.3804-R_MCTC_1.0.WR.mips.337823.180913
ETP Proxy: SS_mcTCMCU-10.1.11.3804-R_MCTC_1.0.WR.mips.337823.180913
root@MCU-0 [scrooge] >

97
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC call flow
CS-Data call related issues can be investigated service terminal commands in
ETPC and MCU.
In ETPC the detpcs command tells the info about ongoing calls
duddl and dudul can be used to investigate that the UDP dataflow is
ongoing.
Ongoing calls in ETPC
MCETPC/0>detpcs
---------------------------------------------------------------> Command detpcs executed on [18/6/2014 23:26:44:729574]
<--------------------------------------------------------------****************************************************************
*
Disclaimer: Running this command may affect the system
performance, therefore it is recommended that it should be used
only under very special conditions and consciously.
****************************************************************
*
mcETPC Total Running Calls = 2

98
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Ongoing calls in ETPC


MCETPC/0>duddl
UDP Packets passed to Apps for DL cores are: 115634
UDP Packets received from Apps for DL cores are: 108005
MCETPC/0>dudul
UDP Packets passed to Apps for UL cores are: 116069
UDP Packets received from Apps for UL cores are: 115995
Commands need to be given several times in sequence to check
that the values are increasing.

mcTC Troubleshooting
mcTC call flow
Command detpcs all gives more details from ongoing calls. IPs in mcBSC
side
Call eg.
1 Info:
Call 2 Info:
---------------------------------------------------Call ID
:: 0x54/84
MGW TLA
:: 10.8.3.86/63266
MCETPC PORT
:: 49320
ETME TLA [0]
:: 10.255.165.180/49152
ETP_CALL_ID[0]
:: 0x14
...
ETMA TLA [0]
:: 10.255.165.84/65534
...
NO OF CHANNELS
:: 1
PAYLOAD TYPE
:: CODEC_DATA
TCU ID
:: 0
DSP ID
:: 1
CORE ID
:: 0
ATER FLOW ID
:: 65792
AoIP FLOW ID
:: 98560
Call State
:: MCETPC_CHM_STATE_ACTIVE
LS Status
:: 0
LI Status
:: 0
Call Type
:: NORMAL_CALL
BCSU Index/Proc ID :: 0x4f74/0x4e0
BSC ID
:: 0x1
Stream ID
:: 0
Bsc_sw_version
:: 1
HO Active
:: NO
99
1/10/17
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

---------------------------------------------------Call ID
:: 0x55/85
MGW TLA
:: 10.8.3.86/63268
MCETPC PORT
:: 49322
ETME TLA [0]
:: 10.255.165.180/49152
ETP_CALL_ID[0]
:: 0x18
...
ETMA TLA [0]
:: 10.255.165.85/65534
...
NO OF CHANNELS
:: 1
PAYLOAD TYPE
:: CODEC_DATA
TCU ID
:: 0
DSP ID
:: 1
CORE ID
:: 0
ATER FLOW ID
:: 65793
AoIP FLOW ID
:: 98561
Call State
:: MCETPC_CHM_STATE_ACTIVE
LS Status
:: 0
LI Status
:: 0
Call Type
:: NORMAL_CALL
BCSU Index/Proc ID :: 0x4f75/0x534
BSC ID
:: 0x1
Stream ID
:: 0
Bsc_sw_version
:: 1
HO Active
:: NO

mcTC Troubleshooting
mcTC call flow
In MCU the mctccli can be used to investigate ongoing calls in TCSU level.
root@MCU-0 [scrooge]
TCSU: 1
Call ID

> mctccli gettcsuactivecalls tcsu 1

Number of active calls: 2


dDTX
Codec type

84
OFF
CODEC_CSD_TRAU_FRAME
85
OFF
CODEC_CSD_TRAU_FRAME
root@MCU-0 [scrooge] >

Ansi env
A_LAW
A_LAW

root@MCU-0 [scrooge] > mctccli gettcsucallstatus tcsu 1


+-----------------------------------------------------------------+
TCSU
1 call status
+-----------------------------------------------------------------+
Call_ID
|
ETP
|
MCU
|
TCU
|
84
|
*
|
*
|
*
|
85
|
*
|
*
|
*
|
86
|
|
|
|
87
|
|
|
|
88
|
|
|
|
10
1/10/17
0
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

Getcallinfo mctccli command collects info from ETPC and TCU


root@MCU-0 [scrooge] > mctccli getcallinfo callid 84
Callinfo requests CallId 84
getcallinfo 84 ---------------------------------------------CallId 84 parameters
------------------------------------------------------------******************
On ETP
*************************
Call 1 Info:
---------------------------------------------------Call ID
:: 0x54/84
--More-MGW TLA
MCETPC PORT
ETME TLA [0]

:: 10.8.3.86/63266
:: 49320
:: 10.255.165.180/49152

******************
On TCU
*************************
| CALL_ID =
84
| Application Status
|
+---------------------------------+-------------------------------+
| PARAMETER
| Ater Value |
A Value | Type
|
+---------------------------------+------------+------------+-----+
| elapsed_time....................|
1567 |
1567 | sec
|
| dtx_status......................|
3 |
0 | Hex
|

mcTC Troubleshooting
mcTC alarms
Alarms from mcTC are visible in master mcBSC.
< AHO:MCTC,;
LOADING PROGRAM VERSION 6.7-0
mcBSC

mcKOFF

2014-06-19

15:07:09

ALARMS CURRENTLY ON
**

mcKOFF
MCTC-0
SWITCH
2014-06-19
ALARM MCTC-0
DPALAR
(0256) 3597 MCTC FIELD-REPLACEABLE UNIT UNAVAILABLE
70283d

03:29:50.90

mcKOFF
MCTC-0
ALARM MCTC-0
(0257) 3594 MCTC PLATFORM ALARM
70159d

03:30:07.09

SWITCH
DPALAR

2014-06-19

END OF ALARMS CURRENTLY ON

10
1/10/17
1
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC alarms
Internally in mcTC alarms can be checked from MCU fsclish shell
Show alarm active prints the active alarms:
root@MCU-0 [scrooge] > show alarm active
Alarm ID| Specific Problem | Alarm Text | Clearing Info| Event Type Name| Alarm Time
62109
| 70159 | MANAGED OBJECT FAILED | 1 | x2 | 2014-06-19 03:30:07:090 EEST
62108
| 70283 | FIELD-REPLACEABLE UNIT UNAVAILABLE | 1 | x5 | 2014-06-19 03:29:50:900 EEST
root@MCU-0 [scrooge] >
Show alarm history prints the alarm history:
root@MCU-0 [scrooge] > show alarm history
Alarm History: List of Alarm Events
Alarm ID | Specific Problem | Alarm Text | Clearing Info
62084 | 70186 | CLUSTER OPERATION INITIATED BY OPERATOR |
...

|
1

Event Type N
| x4 | 20

Alarm printout can be filtered with problem type for example:


root@MCU-0 [scrooge] > show alarm history filter-by specific-problem 70283
Alarm History: List of Alarm Events
Alarm ID
62108 |

| Specific Problem | Alarm Text | Clearing Info | Event Type Name | Alarm Time
70283 | FIELD-REPLACEABLE UNIT UNAVAILABLE | 1 | x5 | 2014-06-19 03:29:50:900 EEST

Detailed mapping of mcBSC and mcTC alarms and problem causes are
described in System Maintenance in BSC, mcBSC, and mcTC DN0975145
10
1/10/17
2
Confidential

Nokia 2014 - File Name - Version - Creator - DocID

mcTC Troubleshooting
mcTC debug scripts & printouts
BCN level debug script is available in mcTC as well.
Additionally in ETPC there is all_debug_info command which prints snapshot
from most important areas from ETPC. (note that all_debug_info command
might causes momentary load peak to the ETPC)

10
1/10/17
3
Confidential

Nokia 2014 - File Name - Version - Creator - DocID