Sie sind auf Seite 1von 10

Final Report

Case Id: NA05837107


TBF congestion while maximum configured TBF/TS not reached

Issue description:
We noticed a problem of TBF congestion in UL/DL (counters related: c072123, c072124,
c072079, c072080) even the max number of configured TBF/TS is not reached (9 for DL and 6/7 for UL).
This problem affects few sectors in the network and isnt very recurrent.

Challenges:
The collection of logs is a hard task since the problem is sporadic. LOGs were collected many times on:
-

TUN3028S1
TUN3249S3
BAR 3098S3
TUN3190S3

Traces were collected on many others sectors (onsite and remote traces), but the problem was resolved
at the time of collection (on the next hour based on KPIs), which make collected traces unusable.
The list of required LOGs was not fixed by Technical support from the beginning: each time, new LOGs
were needed to elaborate new investigation fields.

Final required LOGs list:

Abis trace (onsite)


Raw measurement (last 72 hours)
BSClogcol: basic BSC logs
PCU BMT: messages monitoring in PCU
OFL logs: internal DSP communication in PCU.

The biggest challenge is to have all these LOGs synchronized especially ABIS traces and OFL logs. Abis
traces need onsite coordination and access permission; OFL logs need specific configuration in BSC switch
to have mirrored port to be able to redirect PCU messages flow to local machine (This configuration is
available only on Charguia BSCs).
All these previous constraints have made the case handling heavy and takes a long time.

Issue analysis:
TUN3028S1:
For BTS TUN3028S1, there is a reject even in time of max TBF per TS is not yet reached. We can see that
from below tables and graphs:
Row Labels
10.21.2015
12:00:00
10.21.2015
13:00:00
10.21.2015
14:00:00

MAX_TBFS_PER_TSL_UL (c072103)

Max configured_TBFS_PER_TSL_UL

NO_RADIO_RES_AVA_UL_TBF (c072079)

28

46

80

Row Labels
10.21.2015
12:00:00
10.21.2015
13:00:00

MAX_TBFS_PER_TSL_DL (c072104)

MAX_Configured_TBFS_PER_TSL_DL

Average of NO_RADIO_RES_AVA_DL_TBF (c072080)

86

220

The blue line is showing the number of UL TBF per TSL, the red one is showing trend of counter C072079
(NO_RADIO_RES_AVA_UL_TBF). We can see, in circuled, some records of UL TBF rejects even the max
confiured UL TBF per TSL is not yet reached.
We can see also the fact that the problem is sporadic : it happens just in some hours and it can be resolved
at any time.

The same behavior is seen in DL direction:

TUN3190S3
In previous example, R&D wasnt able to identify a clear root cause since collected logs werent enough
and advanced levels of message flow in PCU were needed. Only OFL LOGs could contain these flow levels.
BTS TUN3190S3 (which is under BSCCHR05) was suufering from the issue of TBF congestion whimaximum
configured TBF/TS not reached. All needed LOGs were collected.

Below code flow scenario have occurred and causes pegging counters of NO RADIO RESOURCE
UNAVAILABLE. From provided latest logs, we are able to see now (with OFL logs) the same scenario has
been occurred and due to that NO RADIO RESOURCE failure counter got incremented.
Code flow scenario:
imsgw_boolean_t pdm_bts_acc_tech_match if access technology not supported that particular BTSs will not add into Eligible BCCH BTS List
for all for BTS in that corresponding segment (p_seg->total_num_bts; count ++)
|
void pdm_add_bts_for_access_tech

Go through all BTSs in particular segment

void pdm_add_bts_for_access_tech(bts_eligible_list, p_ms, direction,


bts_selection_mode, acc_tech); ( will go through all MS RAC Capabilities from ms RAC info and will build bts_eligible
list )
|
Imsgw_boolean_t pdm_select_bts()
hen
will return failures
Else
LOGT_DETAIL_2 (IMSGW_PDM_APP_ID, LOG_APPLICATION,
("\npdm_select_bts: For ms.tlli:[0x%x]/%u, ms.seg_id:%u, \n"
"direction:%u, bts_list_1 is empty. BTS SEL Cause = %s, "
"BTS selection fail cause:%u\n",
p_ms->ms_id.tlli, p_ms->ms_id.tlli, p_ms->ms_id.seg_id, direction,
g_pdm_selection_cause_names[cause], g_bts_sel_fail_due_max_tbf));

LOGT_DETAIL_3 (IMSGW_PDM_APP_ID, LOG_APPLICATION,("Exit function "


"pdm_select_bts with IMSGW_FAILURE \n"));

M_PDM_RESET_DLDC_BTS_INFO();

return (IMSGW_FAILURE);

The same above occurred in LOGs. There are plenty of BTS selection failures due to below reasons. All
will definitely lead to increment the counter values.
One of prints I have pasted below:
[Logging Trace][..\..\pq2\pdm\res_mgr\src\pdm_resmgr_bts_sel.c][1623][17][8/1/2016
10:38:37:175417][][29369]
pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0

Line 282146: direction:0, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0
Line 323424: direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0
Line 883229: direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0
Line 1023559: direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0

Line 1253504: direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=0, BTS selection fail cause:0
Line 1444004: direction:2, bts_list_1 is empty. BTS SEL Cause = PDM_REALLOC, rlc_req=1, BTS selection fail cause:0

Only one specific MS- IMSI has caused this kind of failures. We can see in below LOGs extract all IMSI
details:

IMSI Details:
/**********************************************************************/
[Logging Trace][..\..\pq2\pdm\pdm_mgr\src\pdm_utils.c][15013][17][8/1/2016
10:38:38:390384][][45348]
MS 0x2049f0b0 MS State [0x1] old state:[PDM_ULTBF_ACTIVE]
new:[PDM_ULTBF_NORMAL_REL_IN_PROGRESS] opp:[PDM_DLTBF_NO_STATE]
tlli:[0xe32396c7]/3810760391 seg_id:1909 IMSI:1/605030572353015ul_tbf/o:0x4/0x4
dl_tbf/o:0xffff/0xffff

/*********************************************************************/
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,

PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,


PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_1:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_10:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_11:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_12:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_12:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_12:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_13:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_13:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_2:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_21:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_22:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_22:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_22:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_22:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_23:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,

PCU_LOGS_23:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,


PCU_LOGS_23:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_24:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_24:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_24:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_25:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_27:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_27:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_27:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_27:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_28:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_29:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_3:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_30:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_31:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_31:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_31:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_31:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_31:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,

PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,


PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_4:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_5:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_6:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_7:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,
PCU_LOGS_8:pdm_select_bts: For ms.tlli:[0xe32396c7]/3810760391, ms.seg_id:1909,

/**********************************************************************/
We can monitor this IMSI:1/605030572353015 and try to find out what kind of MS is this.

Conclusion:
At the time of PDM_REALLO during admission control phase, PDM will do BTS selection based on received
MS-RAC info form SGSN (our case SGSN) or MS. At the time of selection phase if received MS-RAC info is
not supported by selected BTSs in corresponding segment then there will be immediate rejection . This
PDM is triggering NO RADIO RESOURCE UNAVAILABLE counters along with TBF release.
It is confirmed that there is no SW issue, and this problem is related to some specific UEs which
sporadically causes the counters trigger (c072123, c072124, c072079, c072080)

10

Das könnte Ihnen auch gefallen