To analyze Response Time we need to know:

total time spent on CPU

total time spent on Waits
Response Time = CPU Time + Total Wait Time

Wait Event Statistics

Wait events are statistics that are incremented by a session to indicate that it had to
wait for an event to complete before being able to continue processing. When a session
has to wait while processing a user request, the AWR records the wait using one of a
set of predefined wait events that are then grouped into wait classes. Wait event data
reveals various symptoms of problems that might be impacting performance, such as
latch contention, buffer contention, and I/O contention

If time spent on wait is significant compared to CPU time then we need to analyze wait
event.

The Automatic Workload Repository (AWR) automates

database statistics gathering by collecting, processing, and maintaining performance
statistics for database problem detection and self-tuning purposes. By default, this
process is repeated every hour, and the result is called an AWR snapshot. The delta
values captured by the snapshot represent the changes for each statistic over the time
period. Statistics gathered by the AWR are queried from memory. The gathered data
can be displayed in both reports and views.

Database tuning is performed in two phases: proactively and reactively

Look at Top 5 Timed Events

From AWR Report

 Event Time(s) % Total Call Time
CPU time 46,373 44.2
Backup: sbtwrite2 13,954 13.3
io done 13,596 13.0
enq: TX - row lock
10,051 9.6
contention
control file sequential read 8,168 7.8

From EM:
CPU bottlenecks:

Is the application performing poorly because the system is CPU bound?

Monitoring CPU Utilization:

Utilization should not be over 90%.

Run Queue shouldn’t be over than 2x number of processors

Follow ‘Diagnosing High CPU Utilization’ Note 164768.1 to define a

procedure for diagnosing high CPU utilization by Oracle

Waits related to memory structures:
Are the Oracle memory structures—such as the SGA, PGA, and buffer
cache—adequately sized?

First we will check the system memory:
If this option is not working then we can use the OSWatcher ( No graphs)
Or Linux monitor tools such as ganeria

From AWR:

Begin End
Buffer Cache: 448M 448M Std Block Size: 8K
Shared Pool Size: 960M 960M Log Buffer: 14,356K

Shared Pool Statistics

Memory Usage %: 80.05 83.77

Should be between 75 – 90 %

Buffer Hit %: 99.22 In-memory Sort %: 100.00

Library Hit %: 99.44 Soft Parse %: 98.59

Should be over 90%

Waits related to I/O:

From EM:

Less than 10 ms is acceptable, normaly

ASM script Metalink note.

AWR Tablespace IO Stats

Av Av Av Av Buffer Av Buf
Tablespace Reads Writes
Reads/s Rd(ms) Blks/Rd Writes/s Waits Wt(ms)
PROJ_LARGE 329 0 2.10 1.00 359 0 0 0.00
SYSAUX 299 0 3.58 1.05 253 0 0 0.00
UNDOTBS1 2 0 0.00 1.00 435 0 0 0.00
PROJ_XLARGE 235 0 0.00 1.00 75 0 0 0.00
SYSTEM 206 0 4.56 1.00 39 0 0 0.0

Waits relate to Suboptimal use of Oracle Database by the application:
establishing new database connection repeatedly, 
AWR:
Logons: 2.68/s
 Review the listener log:
Will show the time and module.
excessive SQL parsing, 

From AWR:
% Non-Parse CPU: 92.26
Should be over 90%

and highlevel of contention for a small amount of data

buffer busy wait
Need to identify the area for contention

waits related to Concurrency issues:
A high degree of concurrent activities might result in
contention for shared resources that can manifest in the forms of locks or waits for
latches.

Interconnect:

Statspack latency name Lower bound (ms) Typical (ms) Upper bound (ms)
Ave receive time for CR block 0.3 3 12
Ave receive time for current block 0.3 8 30

From AWR:
Avg global cache cr block receive time (ms): 3.7
Avg global cache current block receive time (ms): 7.1

Avg. global enqueue get time acceptable less than 10ms

From AWR:
Avg global enqueue get time (ms): 2.7

OSWatcher checking the network card statistics,

OS to check the dropped frames.

Degradation of database performance over time:
Compare our results with the baseline and point out any changes and try to found out.
Resource intensive SQL( logical reads , execution plan , execution time , …)
CPU utilization
Memory utlization
I/O response time
RAC interconnect