ADDM Report for Task 'TASK_180740'

---------------------------------Analysis Period
--------------AWR snapshot range from 59797 to 59801.
Time period starts at 06-NOV-15 09.30.12 AM
Time period ends at 06-NOV-15 10.30.32 AM
Analysis Target
--------------Database 'MABDB' with DB ID 1027027449.
Database version 11.2.0.3.0.
ADDM performed an analysis of instance mabdb2, numbered 2 and hosted at
db02-rac02.
Activity During the Analysis Period
----------------------------------Total database time was 8711 seconds.
The average number of active sessions was 2.41.
Summary of Findings
------------------Description
s

Active Sessions

Recommendation

Percent of Activity
---------------------------------------- ------------------- -------------1
2
3
4
5
6

Top SQL Statements

1.12 | 46.51
Top Segments by "User I/O" and "Cluster" .57 | 23.56
Global Cache Messaging
.29 | 11.85
PL/SQL Execution
.13 | 5.35
Buffer Busy - Hot Objects
.06 | 2.32
Buffer Busy - Hot Block
.03 | 1.43

5
3
1
3
0
2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
---------------------------Finding 1: Top SQL Statements
Impact is 1.12 active sessions, 46.51% of total activity.
--------------------------------------------------------SQL statements consuming significant database time were found. These
statements offer a good opportunity for performance improvement.
Recommendation 1: SQL Tuning
Estimated benefit is .56 active sessions, 23.37% of total activity.
------------------------------------------------------------------Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"gb8dbmdrxwy9q".
Related Object
SQL statement with SQL_ID gb8dbmdrxwy9q.
Select trn_ref_no,AC_ENTRY_SR_NO,VALUE_DT as TRN_DT,TRN_CODE, AC_No,
(maturity_date-int_start_date) as Days, Case When
DRCR_IND=:"SYS_B_00" then DRCR_MNT * (-:"SYS_B_01") end WDL,Case

When DRCR_IND=:"SYS_B_02" then DRCR_MNT end DEP, Running_Bal, AC_NO

from (select ac.trn_ref_no,ac.AC_NO,ac.ac_ccy,ac.VALUE_DT,ac.AC_BRANC
H,ac.DRCR_IND,ac.LCY_AMOUNT,ac.FCY_AMOUNT,ac.AC_ENTRY_SR_NO,ac.extern
al_ref_no,ac.cust_gl,ac.TRN_CODE,ac.AMOUNT_TAG,ac.RELATED_ACCOUNT,ac.
event,sum(decode(ac.DRCR_IND,:"SYS_B_03",ac.LCY_AMOUNT,:"SYS_B_04",-(
ac.LCY_AMOUNT))) over ( partition by
substr(ac.AC_NO,:"SYS_B_05",:"SYS_B_06") order by ac.AC_ENTRY_SR_NO
range between unbounded preceding and current row )
running_bal,decode(ac.DRCR_IND,:"SYS_B_07",ac.LCY_AMOUNT,:"SYS_B_08",
-(ac.LCY_AMOUNT)) drcr_mnt from Acvws_All_Ac_Entries ac where
substr(ac_no,:"SYS_B_09",:"SYS_B_10") = :"SYS_B_11" and
substr(ac_no,:"SYS_B_12",:"SYS_B_13") = :"SYS_B_14" and
substr(ac_no,:"SYS_B_15",:"SYS_B_16")=:"SYS_B_17" and
ac.AC_ENTRY_SR_NO > :"SYS_B_18" group by ac.AC_NO, ac.VALUE_DT,
ac.ac_ccy, ac.AC_BRANCH, ac.DRCR_IND,ac.LCY_AMOUNT, ac.FCY_AMOUNT,
ac.trn_ref_no, ac.cust_gl, ac.AMOUNT_TAG, ac.event,
ac.RELATED_ACCOUNT, ac.TRN_CODE,ac.AC_ENTRY_SR_NO,ac.external_ref_no
order by ac.AC_ENTRY_SR_NO) INNER JOIN ictm_acc ON AC_No= ACC Where
AC_ENTRY_SR_NO >= :"SYS_B_19" and trn_ref_no not in (select
trn_ref_no from acvws_all_ac_entries where event=:"SYS_B_20") order
by AC_ENTRY_SR_NO
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "gb8dbmdrxwy9q" was executed 40 times and had
an average elapsed time of 51 seconds.
Recommendation 2: SQL Tuning
Estimated benefit is .24 active sessions, 10% of total activity.
---------------------------------------------------------------Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"69r6dt63qzmhy".
Related Object
SQL statement with SQL_ID 69r6dt63qzmhy.
SELECT Count(XrefId) FROM ( SELECT DISTINCT F.XREFID FROM
FBTB_TXNLOG_MASTER F ,FBTB_FUNCTION_DESCRIPTION D
,FBTB_TXNLOG_DETAILS T WHERE (D.FUNCTION_ID(+) = F.FUNCTIONID) AND
upper(F.BRANCHCODE) IN (SELECT BRANCH_CODE FROM FBTB_USER_ROLE WHERE
USERID= :1 ) AND F.FUNCTIONID IN (select FUNCTION_ID from FBTB_MENU
where nvl(multibranch_access,:"SYS_B_00")=:"SYS_B_01") AND F.XREFID =
T.XREFID AND F.BRANCHCODE = T.BRANCHCODE AND T.STAGESTATUS IN
(:"SYS_B_02", :"SYS_B_03") AND T.CHECKERID = :2 AND F.POSTINGDATE
BETWEEN TO_DATE( :3 , :"SYS_B_04") AND TO_DATE( :4 , :"SYS_B_05")
AND upper(F.BRANCHCODE) like upper(:5 ) UNION SELECT DISTINCT
F.XREFID FROM FBTB_TXNLOG_MASTER F ,FBTB_FUNCTION_DESCRIPTION D
,FBTB_TXNLOG_DETAILS T WHERE (D.FUNCTION_ID(+) = F.FUNCTIONID) AND
F.BRANCHCODE = :6 AND F.FUNCTIONID IN (select FUNCTION_ID from
FBTB_MENU where nvl(multibranch_access,:"SYS_B_06")=:"SYS_B_07") AND
F.XREFID = T.XREFID AND F.BRANCHCODE = T.BRANCHCODE AND
T.STAGESTATUS IN (:"SYS_B_08", :"SYS_B_09") AND T.CHECKERID = :7
AND F.POSTINGDATE BETWEEN TO_DATE( :8 , :"SYS_B_10") AND TO_DATE( :9
, :"SYS_B_11") AND upper(F.BRANCHCODE) like upper(:10 ))
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.

This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "69r6dt63qzmhy" was executed 211 times and had
an average elapsed time of 3.8 seconds.
Rationale
At least 2 distinct execution plans were utilized for this SQL statement
during the analysis period.
Recommendation 3: SQL Tuning
Estimated benefit is .13 active sessions, 5.35% of total activity.
-----------------------------------------------------------------Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"5rjhjr608q99q".
Related Object
SQL statement with SQL_ID 5rjhjr608q99q.
select * from
(
select a.trn_ref_no,a.ac_no,a.instrument_code,a.dr,a.cr,a.bal,b.benef
_name,
b.identifier_value_1, trn_dt,b.issuing_branch, PO_Type, from_branch
from
(select
trn_ref_no,
ac_no,
instrument_code,
Sum(case when drcr_ind=:"SYS_B_00" then lcy_amount end) DR,
Sum(case when drcr_ind=:"SYS_B_01" then lcy_amount end) CR,
Sum(case when drcr_ind=:"SYS_B_02" then -lcy_amount else lcy_amount
end) Bal,
(case when event=:"SYS_B_03" then trn_dt end) as trn_dt
from acvws_all_ac_entries
where substr(trn_ref_no,:"SYS_B_04",:"SYS_B_05")= :P_BranchCode
and ac_no = :P_Type
and event <> :"SYS_B_06"
/*and trn_ref_no not in (select trn_ref_no from acvws_all_ac_entries
where event ='REVR')*/
group by trn_ref_no, ac_no, instrument_code, trn_dt, event
having Sum(case when drcr_ind=:"SYS_B_07" then -lcy_amount else
lcy_amount end)>:"SYS_B_08"
) A
inner join
(select contract_ref_no,issuing_branch, payable_branch, benef_name,
identifier_value_1
from istms_instr_txn a
where (a.instr_type like :"SYS_B_09" or a.instr_type like
:"SYS_B_10")
and a.instr_stat <> :"SYS_B_11"
and A.Expiry_Date >= trunc(sysdate)
--and a.record_stat = 'O'
and a.instr_stat !=:"SYS_B_12"
) B
on A.trn_ref_no = B.contract_ref_no
left join
(select contract_ref_no, field_val_1 as PO_Type, field_val_2 as
from_branch from cstms_contract_userdef_fields)D

on A.trn_ref_no = D.contract_ref_no
union all
select a.trn_ref_no,a.ac_no,a.instrument_code,a.dr,a.cr,a.bal,b.benef
_name, b.identifier_value_1, trn_dt,
b.issuing_branch,:"SYS_B_13" as PO_Type, from_branch from
(select
trn_ref_no,
ac_no,
instrument_code,
Sum(case when drcr_ind=:"SYS_B_14" then lcy_amount end) DR,
Sum(case when drcr_ind=:"SYS_B_15" then lcy_amount end) CR,
Sum(case when drcr_ind=:"SYS_B_16" then -lcy_amount else lcy_amount
end) Bal,
(case when event=:"SYS_B_17" then trn_dt end) as trn_dt
from acvws_all_ac_entries
where /*substr(trn_ref_no,1,3)= :P_BranchCode and*/
ac_no = :P_Type
and event <> :"SYS_B_18"
/*and trn_ref_no not in (select trn_ref_no from acvws_all_ac_entries
where event ='REVR')*/
group by trn_ref_no, ac_no, instrument_code, trn_dt, event
having Sum(case when drcr_ind=:"SYS_B_19" then -lcy_amount else
lcy_amount end)>:"SYS_B_20"
) A
inner join
(select contract_ref_no, payable_branch, benef_name,
identifier_value_1,case
when(substr(narrative,:"SYS_B_21",:"SYS_B_22")in(select branch_code
from sttm_branch)) then substr(narrative,:"SYS_B_23",:"SYS_B_24")
else issuing_branch end as issuing_branch
from istms_instr_txn a
where a.instr_type like :"SYS_B_25"
and payable_branch=:P_BranchCode
and a.instr_stat <> :"SYS_B_26"
and A.Expiry_Date >= trunc(sysdate)
--and a.record_stat = 'O'
and a.instr_stat !=:"SYS_B_27"
) B
on A.trn_ref_no = B.contract_ref_no
left join
(select branch_code, branch_name as from_branch from sttms_branch) br
on issuing_branch = branch_code
)E
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "5rjhjr608q99q" was executed 12 times and had
an average elapsed time of 28 seconds.
Rationale
At least 2 distinct execution plans were utilized for this SQL statement
during the analysis period.
Recommendation 4: SQL Tuning
Estimated benefit is .1 active sessions, 4.19% of total activity.
-----------------------------------------------------------------

Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"4sq7rkky2stn2".
Related Object
SQL statement with SQL_ID 4sq7rkky2stn2.
select nvl(DRAWING_POWER,:"SYS_B_0") as
DRAWING_POWER,nvl(ACY_AVL_BAL,:"SYS_B_1") AS AVL_BAL from
stvw_ac_bal_sum a where a.cust_ac_no=:"SYS_B_2"
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "4sq7rkky2stn2" was executed 87 times and had
an average elapsed time of 4 seconds.
Recommendation 5: SQL Tuning
Estimated benefit is .09 active sessions, 3.6% of total activity.
----------------------------------------------------------------Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"9hh9ncmn8q6ar".
Related Object
SQL statement with SQL_ID 9hh9ncmn8q6ar.
SELECT (nvl(DRAWING_POWER,:"SYS_B_0") + nvl(ACY_AVL_BAL,:"SYS_B_1"))
AS AVL_BAL FROM stvw_ac_bal_sum a WHERE a.cust_ac_no=:"SYS_B_2"
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "9hh9ncmn8q6ar" was executed 180 times and had
an average elapsed time of 1.6 seconds.
Finding 2: Top Segments by "User I/O" and "Cluster"
Impact is .57 active sessions, 23.56% of total activity.
-------------------------------------------------------Individual database segments responsible for significant "User I/O" and
"Cluster" waits were found.
Recommendation 1: Segment Tuning
Estimated benefit is .2 active sessions, 8.32% of total activity.
----------------------------------------------------------------Action
Run "Segment Advisor" on TABLE "FCUBSPROD.ISTM_INSTR_TXN" with object ID
511436.
Related Object
Database object with ID 511436.
Action
Investigate application logic involving I/O on TABLE
"FCUBSPROD.ISTM_INSTR_TXN" with object ID 511436.
Related Object
Database object with ID 511436.

Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the SELECT statement with
SQL_ID "5rjhjr608q99q" is responsible for 48% of "User I/O" and
"Cluster" waits for this segment.
Rationale
The I/O usage statistics for the object are: 0 full object scans,
2027407 physical reads, 205 physical writes and 2027379 direct reads.
Recommendation 2: Segment Tuning
Estimated benefit is .19 active sessions, 7.97% of total activity.
-----------------------------------------------------------------Action
Run "Segment Advisor" on TABLE "FCUBSPROD.FBTB_TXNLOG_DETAILS" with
object ID 514447.
Related Object
Database object with ID 514447.
Action
Investigate application logic involving I/O on TABLE
"FCUBSPROD.FBTB_TXNLOG_DETAILS" with object ID 514447.
Related Object
Database object with ID 514447.
Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the SELECT statement with
SQL_ID "69r6dt63qzmhy" is responsible for 100% of "User I/O" and
"Cluster" waits for this segment.
Rationale
The I/O usage statistics for the object are: 25 full object scans,
9908907 physical reads, 1134 physical writes and 9907900 direct reads.
Recommendation 3: Segment Tuning
Estimated benefit is .17 active sessions, 7.27% of total activity.
-----------------------------------------------------------------Action
Run "Segment Advisor" on TABLE "FCUBSPROD.ACTB_HISTORY" with object ID
508444.
Related Object
Database object with ID 508444.
Action
Investigate application logic involving I/O on TABLE
"FCUBSPROD.ACTB_HISTORY" with object ID 508444.
Related Object
Database object with ID 508444.
Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the SELECT statement with
SQL_ID "gb8dbmdrxwy9q" is responsible for 100% of "User I/O" and
"Cluster" waits for this segment.
Rationale
The I/O usage statistics for the object are: 40 full object scans,
725195 physical reads, 0 physical writes and 724584 direct reads.
Symptoms That Led to the Finding:
--------------------------------Wait class "User I/O" was consuming significant database time.
Impact is .59 active sessions, 24.36% of total activity.
Finding 3: Global Cache Messaging

Impact is .29 active sessions, 11.85% of total activity.

-------------------------------------------------------Inter-instance messaging was consuming significant database time on this
instance.
Recommendation 1: Application Analysis
Estimated benefit is .29 active sessions, 11.85% of total activity.
------------------------------------------------------------------Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant time on Cluster waits. For example, the SELECT statement
with SQL_ID "gb8dbmdrxwy9q" is responsible for 64% of Cluster wait
during the analysis period.
Symptoms That Led to the Finding:
--------------------------------Wait class "Cluster" was consuming significant database time.
Impact is .29 active sessions, 11.85% of total activity.
Finding 4: PL/SQL Execution
Impact is .13 active sessions, 5.35% of total activity.
------------------------------------------------------PL/SQL execution consumed significant database time.
Recommendation 1: SQL Tuning
Estimated benefit is .07 active sessions, 3.02% of total activity.
-----------------------------------------------------------------Action
Tune the entry point PL/SQL
"FCUBSPROD.GWPKS_SERVICE_ROUTER.PR_PROCESS_REQ_MSG#1" of type "PACKAGE"
and ID 518654. Refer to the PL/SQL documentation for addition
information.
Recommendation 2: SQL Tuning
Estimated benefit is .03 active sessions, 1.16% of total activity.
-----------------------------------------------------------------Action
Tune the entry point PL/SQL
"FCUBSPROD.GWPKS_MSG_ROUTER.PR_PROCESS_REQ_MSG" of type "PACKAGE" and ID
518455. Refer to the PL/SQL documentation for addition information.
Recommendation 3: SQL Tuning
Estimated benefit is .03 active sessions, 1.16% of total activity.
-----------------------------------------------------------------Action
Tune the entry point PL/SQL
"FCUBSPROD.GWPKS_NOTIFICATION_FCJ.PR_NOTIF_CALLBACK" of type "PACKAGE"
and ID 523593. Refer to the PL/SQL documentation for addition
information.
Finding 5: Buffer Busy - Hot Objects
Impact is .06 active sessions, 2.32% of total activity.
------------------------------------------------------Read and write contention on database blocks was consuming significant
database time.
No recommendations are available.

Symptoms That Led to the Finding:

--------------------------------Read and write contention on database blocks was consuming significant
database time.
Impact is .06 active sessions, 2.32% of total activity.
Inter-instance messaging was consuming significant database time on
this instance.
Impact is .29 active sessions, 11.85% of total activity.
Wait class "Cluster" was consuming significant database time.
Impact is .29 active sessions, 11.85% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .05 active sessions, 2.01% of total activity.
Finding 6: Buffer Busy - Hot Block
Impact is .03 active sessions, 1.43% of total activity.
------------------------------------------------------A hot data block with concurrent read and write activity was found. The block
belongs to segment "FCUBSPROD.ISTM_INSTR_TXN" and is block 1518834 in file 8.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 1.43% of total activity.
-----------------------------------------------------------------Action
Investigate application logic to find the cause of high concurrent read
and write activity to the data present in this block.
Related Object
Database block with object number 511436, file number 8 and block
number 1518834.
Recommendation 2: Schema Changes
Estimated benefit is .03 active sessions, 1.43% of total activity.
-----------------------------------------------------------------Action
Consider rebuilding the TABLE "FCUBSPROD.ISTM_INSTR_TXN" with object ID
511436 using a higher value for PCTFREE.
Related Object
Database object with ID 511436.
Symptoms That Led to the Finding:
--------------------------------Read and write contention on database blocks was consuming significant
database time.
Impact is .06 active sessions, 2.32% of total activity.
Inter-instance messaging was consuming significant database time on
this instance.
Impact is .29 active sessions, 11.85% of total activity.
Wait class "Cluster" was consuming significant database time.
Impact is .29 active sessions, 11.85% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .05 active sessions, 2.01% of total activity.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
----------------------

Miscellaneous Information
------------------------Wait class "Application" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
CPU was not a bottleneck for the instance.
Wait class "Network" was not consuming significant database time.
The network latency of the cluster interconnect was within acceptable limits
of 1 milliseconds.
Session connect and disconnect calls were not consuming significant database
time.
Hard parsing of SQL statements was not consuming significant database time.