Resolve ORA-4031 Error in 4 Steps

How to resolve ORA-4031 error
Jaime Figueroa - Advanced Resolution

-1-
TABLE OF CONTENT
INTRODUCTION.........................................................................................................................................2
SHARED POOL OVERVIEW................................................................................................................2
TOOLS AVAILABLE TO COLLECT INFORMATION................................................................2
ORA-4031 CAUSED FROM THE APPLICATION SIDED........................................................2
DATABASE ISSUES CAUSING ERRORS ORA-4031.............................................................2
METHODOLOGY FOR THE DIAGNOSTIC...................................................................................2
CASE STUDIES..........................................................................................................................................2
CASE 1. 4358485.996........................................................................................................................2
CASE 2. 4413750.999........................................................................................................................2
CASE 3. TAR 4383850.996...............................................................................................................2
CASE 4. TAR 4417034.995..............................................................................................................2
CASE 5. TAR 4326100.996................................................................................................................2
APPENDIX A - Important SQLs.....................................................................................................2
1/40

-2-
INTRODUCTION
The error ora-4031 generates a high volume of service requests to Oracle Support. This document has been
created to provide a description of the essentials of this error, details about the method used to diagnose the
error, usage of tools and sqls available to collect information. The document contains different case studies
from service requests where the material of this document was applied.
Although this error is reported for a specific area, the shared pool, the cause of the problem could come from
distinct layers, plsql, streams, row cache, backup and recovery (RMAN), etc. This method will provide a
general approach that will collect the information required for the diagnostic and this is independent of the
layer involved.
2/40

-3-
SHARED POOL OVERVIEW

The memory allocated for the shared pool is divided in heaps that can be composed of different sub heaps.
This is similar to a tablespace that stores many segments. The tablespace will be the heap and the segments
the subheaps. The extents within the segment are the subheaps of the subheap.
The space management is similar, where the goal is to avoid fragmentation, allocating the piece of best size,
coalescing adjacent free space and detecting memory that can be flushed and reused.
A reduced picture of the shared pool is presented in figure 1.1
Free List Buckets
Free Lists
Heap
Descriptor
The free list buckets is a structure of free lists and each list correspond to a specific size. Oracle does a
binary search on the free list sizes to find the appropriate free list. The first bucket that is greater or equal to
the requested size will be returned. We will continue to walk the free list until we find a bucket which points
to a large enough extent of memory.
To get a better idea, see this informatio created by the heap.awk tool, for the free list bucket summary:
Free List Bucket Summary :
Bucket 0 [size=32
] Count= 0
Bucket 1 [size=40
] Count= 443
Bucket 2 [size=48
] Count= 1850
Av.Size=
Av.Size=
Av.Size=
0.00 Max=
40.00 Max=
48.00 Max=
0
40
48
This shows that bucket 1 has 443 chunks of memory where the maximum size is 40bytes and the average is
40bytes. Bucket 2 is a free list of memory chunks with sizes between 40 and 48 bytes.
3/40
Every bucket will have a free list

specified.

-4composed of chunks of memory and the size will be in the range
When the database is started, the free list buckes will show buckets with chunks of large sizes.
Fragmentation or lack of space will show buckets with smaller chunk sizes.
When a chunk of space is freed, it is added to the bucket whose size is less than or equal to the chunks size.
To allocate space:
Find a free list bucket pointing to a linked list of heaps of size requested or larger.
Search the list for best fit
If larger, use what is needed and return the rest
If none found, ask other modules to identify freeable subheaps until enough memory is found
If unsuccessful, signal ora-4031
If successful, reuse or create a new heap descriptor
This works similar to extent allocations within the tablespaces. We first look for best fit. If not found we
may break up a larger chunk into smaller pieces. If nothing is larger, we invoke other modules to identify
memory that is freeable. As we free a heap, we look at its neighbors. If one or both are also free, then we
coalesce. We see if enough memory is available. If not we loop through the process again.
Allocation classes
Memory within shared pool can be allocated as PERMANENT wich remains until parent heap is
freed and is most used for library cache objects. FREEABLE memory is memory that can be reused
and RECREATABLE, memory that if is flushed from the shared pool it can be rebuilt if necessary.
There is not a specific list of structures or elements that can be list as permanent, freeable or
recreatable. After following this document, it will be possible to identify which type of memory is
causing the error ora-4031 and track the module where the memory is allocated.
Error
Error ora-4031 basically has three main arguments, size requested, area and comment.
ORA-4031: unable to allocate <size requested> bytes of shared memory ("area ","comment)
ORA-4031: unable to allocate 2196 bytes of shared memory
("shared pool","JOB$","KGLS heap","KGLS MEM BLOCK")
ORA-4031: unable to allocate 656 bytes of shared memory
(shared pool","select user#,type# from user...","sql area","ctxdef : kkslod")
4/40

-5The argument size requested indicates how much memory was not found in the subheap, which is
reported by the second argument area. The area could be shared pool or large pool.
The last argument comment is used to identify in which area of the code the memory is allocated.
It is important to mention that the area referenced in error ora-4031 could not be the main cause of
the problem, and it is just the victim that did not find the space. The culprit could be in a different
layer. Trace files will help to identify the cause of the problem
5/40

-6-
TOOLS AVAILABLE TO COLLECT INFORMATION

Shared pool is memory subdivided into chunks called heaps. The content of those heaps can be obtained
using certain events, which dump information into a trace file. There are internal tools that read the
content of the trace file and produce a new file, where the information is better organized.
Events
A. Heapdump event
The heapdump event is used to dump memory from different subheaps. Errors ora-4030 are
associated with problems in the pga, uga or cga heaps, and error ora-4031 is related only to problems
with the shared pool or large pool.
To diagnose problems for error ora-4031, event heapdump level 2 is required. The event can be set
on the fly running command alter system set events 4031 trace name heapdump level 2
scope=memory; or it can be set in the parameter files events=4031 trace name heapdump, level 2.
Event heapdump can also be used to get the heapdump at any time attaching to a process using
oradebug:
SQL>oradebug setmypid
SQL>oradebug dump heapdump 2
SQL>oradebug tracefile_name
When dealing with error ora-4031 there is not required dumping other heaps like pga, top call or
current call, so do not use level 13, 3, etc.
Here is a list of all levels available for heapdump event.
HEAP
LEVEL
HEX
Top PGA
Top SGA
Top UGA
Current call
User call
Large Pool
DEC
0x01
0x02
0x04
0x08
0x10
0x20
1
2
4
8
16
32
6/40
WITH CONTENTS
HEX
DEC
0x401
1025
0x802
2050
0x1004
4100
0x2008
8200
0x4010
16400
0x8020
32800

-7The levels are acumulative, so if want to dump sga and large pool, the correct level will be 34. These
levels can be also combined with the special level 536870912. More details are explained below.
10G included diagnostics for error ora-4031, creating a default trace file. Following parameters
control this feature in 10G:
Parameter
Session Value Instance Value
------------------------------ --------------- --------------_4031_dump_bitvec
20479
20479
_4031_max_dumps
100
100
_4031_dump_interval
300
300
_4031_sga_dump_interval
3600
3600
_4031_sga_max_dumps
10
10
In 9iR2 these changes are controled by parameter _4031_sga_dump_interval.
Other level used when using heapdump event is 536870912, which also dumps the 5 largest subheaps
and the 5 largest within each of those. To use this level and dump the shared pool, the command is:
Alter system set events 4031 trace name heapdump level 536870914;
Alter system set events 4031 trace name errorstack level 10;
536870914 = 2 (shared pool) + 536870912 ( dump 5 largest subheaps).
When error ora-4031 is reported in the large pool, heapdump level 32 can be used:
Where
536870944 = 32 (large pool) + 536870912 ( dump 5 largest subheaps).
B. event 10235 level 65536.
It was mentioned in the introduction, that memory in shared pool can be allocated as freeable,
recreatable or permanent. Memory could have a comment that is used to identify the area or the
module where the memory is allocated, and it is reflected in the heapdump or through different
views.
Chunk
Chunk
Chunk
38e0caf48 sz=
38e0cc790 sz=
38e0cdfd8 sz=
6216 R-freeable "library cache " ds=3c6ce6130

6216 R-freeable "library cache " ds=3c6ce6130
40 R-freeable "reserved stoppe"
7/40

-8Chunk
38e0ce000 sz= 4904832 perm
"perm
" alo=4904832
Chunk
38e57b780 sz= 10851616 perm "perm
" al
Figure 2.1. Chunks of memory allocated as permanent without comments.
In Figure 2.1, notice that chunks of type R-freeable have a comment like library cache or reserved
stopped.
By default, memory allocated as permanent do not store comments. In situations where this type of
memory is an issue, it is imposible to identify the module where memory is allocated.
Permanent memory will be released only when the instance is recycled. Leaks on permanent
memory and if it uses more than 35% of the shared pool, the environment will be exposed to errors
ora-4031 or other libry cache problems.
Event 10235 level 65536 will store comments for permanent memory and t can be set on the fly but it
will require a flush of the shared pool. It is better if the instance can be recycled because a flush of
the shared pool does not guarantee a complete flush of the shared pool.
Before 9205, event 10235 at level four was used to store comments but this caused huge performance
degradation. As a result level 65536 was introduced and will cause very light performance
degradation.
Do not get confused by other leves of event 10235 (1,2,3,4) which only can be set at instance level.
Once event 10235 is set and the shared pool has reached is normal usage, a dump of shared pool
needs to be executed.
- Using heapdump and heap.awk
Trace files created for event heapdump can be large and difficult to analyze. Fortunately heap.awk
tool was created to interpret these dumps and present a summary file. This tool will be attached as
part of the document or can be downloaded from Kernel Notes Site or here.
When downloading this tool from the page, edit and replace the Cntrl-M character to avoid getting
errors during the execution.
Sintax:
awk f heap.awk <trace_file_name.trc> > output_file.out
where
trace_file_nam.trc is the trace file created by event heapdump
output_file.out is the name to be created by using the tool.
NOTE: On Solaris platform use nawk.
8/40

-9The trace file will dump information for every subheap from the shared pool. The output file created
by heap.awk will put the information for every subheap, distributed in three principal sections,
described below:
The names used for this section are not standard, are only part of this document and the porpouse if to
provide some type of organization.
Section One. General Space Allocation

Displays all the memory allocated for every subheap (sga heap(1,0, sga heap(2,0), etc) in the format:
DESCRIPTION
Indicates the type of memory allocated. Possible values are free (real free memory),
freeable (can be reused), permaent (only reused recycling the instance) or recreate (can be
reused if is not pinned). R-free and R-freeable belong to the reserved space determined
by parameter shared_pool_reserved_size.
Count
Number of chunks allocated
Sum
Total spaced allocated for each type of memory
Average This is only an estimate (sum/count) that determines the average byte used for every
chunk of memory allocated.
Table 2.1. Description of information displayed in the General Space Allocation section
Type
---> HEAP DUMP heap name="sga heap(1,0)" desc=38002a9b0

Type
Count
Sum
Average
---------------------------------------------------------------------------R-freeable
110
219712
1997.38
R-free
82
67057536
817774.83
perm
225
111658464
496259.84
free
13245
1328584
100.31
freeable
205574
396217864
1927.37
recreate
330378
246845912
747.16
This section is used to determine how is the distribution of the memory in the shared pool, and is
possible to identify which type of memory is consuming most of the space. Generally, larger
allocations of freeable or permanent memory (leaks) are signs of a system that could raise error ora4031.
Also this section provides the real value for free memory because is displayed individually for each
subheap or subpool. Output from v$sgastat displays the value as a total including all the subheaps.
The free memory has to be evaluated for each subheap.
9/40

- 10 Section Two. Breakdown section
In this section, you will find how is distributed or allocated the memory in the subheap. The
comment stored in the shared pool is used to identify the type.
BreakDown
~~~~~~~~~
Type
Count
Sum
Average
-------------------------------------------------------------------------reserved stoppe
80
3200
40.00
Free
13327
68386120
5131.40
library cache
167025
124750280
746.90
sql area
74195
275343400
3711.08
session param v
179
4918920
27480.00
kzull
348
32064
92.14
KGL handles
69276
34363096
496.03
fixed allocatio
80
4480
56.00
PRESENTATION TA 1
2072
2072.00
listener addres
1
56
56.00
KGLS heap
68545
115979192
1692.02
KQR PO
155718
87227112
560.16
table definiti
398
72656
182.55
modification h
64
263680
4120.00
PL/SQL MPCODE
66
183664
2782.79
errors
17
15080
887.06
Like in section one, this section shows how many chunks with a specific comment are allocated
(count), what is the total space (Sum) and what is the average size (Average).
In environments with leaks in permanent allocations, section two will display this memory as Free or
perm type. This is because there is not comment stored in the shared pool for permanent memory.
Event 10235 level 65535 will have to enabled and generate a new heapdump.
The information in this section helps to identify what type of memory is consuming most of the space
(under type column). This is the comment stored within the shared pool for every memory
allocation. Large values under sum column will have to be review as they could be the cause for
errors ora-4031 due to memory leaks. The comment (type column) is used to research in WebIV for
bugs/notes. In tao same value can be used to identify the routines where this type of memory is
allocated. Knowing the name of the routine helps to find events that could be used as workarounds or
to adjust the search in WebIV.
10/40

- 11 Section Three. Free List Bucket Summary
This section will display how the free memory is distributed.
Bucket
Size
Count
Av.Size
Max
DESCRIPTION
The hash table used to store the free memory is divided in buckets. Each bucket
stores allocations of a particular size
Determines the size of the allocations stored in the bucket
Number of free chunks allocated in the bucket
Average size for each chunk allocated in the bucket
Maximum size of the chunk that will be find in the bucket
Table 2.2. Description of columns for the breakdown section
In the following list it can be observed that free memory is available in the first 9 buckets. The
maximum size is in bucket 9 with 104 bytes. There are 396 chunks which give around 40k of free
memory.
The value in section one for free memory was 1328584. Now using the information from section
three, it can be determined that this 1328584 bytes are distributed in very little chunks.
Bucket 0 [size=32
] Count= 0
Bucket 1 [size=40
] Count= 443
Bucket 2 [size=48
] Count= 1850
Bucket 3 [size=56
] Count= 2007
Bucket 4 [size=64
] Count= 588
Bucket 5 [size=72
] Count= 455
Bucket 6 [size=80
] Count= 556
Bucket 7 [size=88
] Count= 744
Bucket 8 [size=96
] Count= 625
Bucket 9 [size=104
] Count= 396
/* Some lines removed on porpouse */
Bucket 65 [size=552
] Count= 0
Bucket 66 [size=560 ] Count= 0
/* Continue until the last bucket.
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
0.00 Max=
40.00 Max=
48.00 Max=
56.00 Max=
64.00 Max=
72.00 Max=
80.00 Max=
88.00 Max=
96.00 Max=
104.00 Max=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
Av.Size=
0.00 Max=
0.00 Max=
0.00 Max=
0.00 Max=
0.00 Max=
0
40
48
56
64
72
80
88
96
104
0
0
0
0
0
When the system presents a similar image to the data referenced above, this is an indication that
system is running with few memory error ora-4031 is emminent. A system with a large amount of
free memory will display large values for the buckets on the bottom of the list.
11/40

- 12 -
ORA-4031 CAUSED FROM THE APPLICATION SIDED

Most cases of error ora-4031 are generated mainly due to problems in the database, caused by a bad
configuration in one of the pools (shared pool, large pool, java pool), changes in the implementation that
introduce more memory usage or introduce an Oracle Bug.
But a bad application can also generate errors 4031 and below is a reference to some examples.
Application not using bind variables.

If there is a high volume of sqls using literals, the cursors wont be shared in the shared pool. The
result is an over usage of space that could produce fragmentation and finally error ora-4031.
How to verify this?
SELECT substr(sql_text,1,90) "SQL",count(*) , sum(executions) "TotExecs"
FROM v$sqlarea
WHERE executions < 5
GROUP BY substr(sql_text,1,90) HAVING count(*) > 30 ORDER BY 2 ;
This query has to be reviewed for each case, particularly the length used in the substr function. When
using literals, many sqls will be very similar, and the the difference will be only in the literal used.
Using the first 90 characters of the sql, in some cases will not display sufficient information when
there is a large statement. The goal is to use the appropiate length that will help the analyst to
identify very similar statements, where the difference is only in the literals.
Here is an example with a slight modification of the query:
SELECT substr(sql_text,1,40) "SQL",
count(*) ,
sum(executions) "TotExecs"
FROM v$sqlarea
WHERE executions < 5 and sql_text like 'select sal%'
GROUP BY substr(sql_text,1,40)
ORDER BY 2 ;
The results:
SQL
COUNT(*) TotExecs
-----------------------------------------------------------------------select sal from emp where empno=7782
1
1
select sal from emp where empno=7839
1
1
1
1
1
1
Notice that reading the first 40 bytes it shows the select is similar and the difference is only in the
literal value. If the length is adjusted to 32 the output is:
12/40

- 13 SQL
COUNT(*) TotExecs
--------------------------------------------------------------------select sal from emp where empno=
4
4
Another key point here is the TotExecs. Under this situation TotExecs will be very close to
COUNT(*), which indicates that every individual statement was executed few times, close to 1.
As a workaround, parameter cursor_sharing can be set to SIMILAR or FORCE. The literals will be
replaced by system generated bind variables and a new sql will be generated. If you found bind
variables with format SYS_B_#, then they are generated by cursor_sharing.
SELECT FROM TABLE WHERE REO_DISBRMT_LNE_ITM.REO_DISBRMT_OID IN
(:"SYS_B_0",:"SYS_B_1",:"SYS_B_2",:"SYS_B_3")
Multiple child cursors

Every child cursor allocates space from the shared pool and is the reason for limiting the available
memory. Normally all the cursors should not generate child cursors and values under 50 for
version_count column (v$sqlarea) is normal. If values go up and there are many cursors with
same behaviour, the system will be affected.
To work on these cases, information returned from v$sqlarea and v$sql_shared_cursor is used.
(Version 10g)
select sa.sql_text,sa.version_count ,ss.*from v$sqlarea sa,v$sql_shared_cursor ss
where sa.address=ss.address and sa.version_count > 50
order by sa.version_count ;
(Version 8,9)
select sa.sql_text,sa.version_count ,ss.*from v$sqlarea sa,v$sql_shared_cursor ss
where sa.address=ss.kghldpar and sa.version_count > 50
order by sa.version_count ;
The query listed above will show cursor with more than 50 children and will display all columns
from view v$sql_shared_cursor. If the value for specific columns is Y, indicates is the reason for
the generation of multiple childs. Table 3.1 has a description of the columns used to identify of
the cause.
This topic was focused on not using bind variables, but notice that there are many causes for
generation of multiple child cursors.
13/40

- 14 DESCRIPTION
The existing child cursor was not fully built (in other
words, it was not optimized)
The SQL type does not match the existing child cursor
The optimizer environment does not match the existing
child cursor
The outlines do not match the existing child cursor
The existing statistics do not match the existing child
cursor
Non-data literal values do not match the existing child
cursor
Security level does not match the existing child cursor
COLUMN
UNBOUND_CURSOR
SQL_TYPE_MISMATCH
OPTIMIZER_MISMATCH
OUTLINE_MISMATCH
STATS_ROW_MISMATCH
LITERAL_MISMATCH
SEC_DEPTH_MISMATCH
EXPLAIN_PLAN_CURSOR
BUFFERED_DML_MISMATCH
PDML_ENV_MISMATCH
PDML
INST_DRTLD_MISMATCH
SLAVE_QC_MISMATCH
TYPECHECK_MISMATCH
AUTH_CHECK_MISMATCH
BIND_MISMATCH
DESCRIBE_MISMATCH
LANGUAGE_MISMATCH
TRANSLATION_MISMATCH
ROW_LEVEL_SEC_MISMATC
H
INSUFF_PRIVS
The child cursor is an explain plan cursor and should not

be shared
Buffered DML does not match the existing child cursor
environment does not match the existing child cursor
Insert direct load does not match the existing child cursor
The existing child cursor is a slave cursor and the new
one was issued by the coordinator (or, the existing child
cursor was issued by the coordinator and the new one is a
slave cursor)
The existing child cursor is not fully optimized
Authorization/translation check failed for the existing
child cursor
The bind metadata does not match the existing child
cursor
The typecheck heap is not present during the describe for
the child cursor
The language handle does not match the existing child
cursor
The base objects of the existing child cursor do not match
The row level security policies do not match
Insufficient privileges on objects referenced by the

existing child cursor
INSUFF_PRIVS_REM
Insufficient privileges on remote objects referenced by
the existing child cursor
REMOTE_TRANS_MISMATCH
The remote base objects of the existing child cursor do
not match
Table 3.1. Columns from v$sql_shared_cursor.
If value is Y, is the reason for generation of child cursors.
14/40

- 15 Normally to track cursor sharing problems, event 10270 is used together with event 10046. It is
good to use when the problem can be reproduced, so the events can be set t the session level:
alter session set events '10046 trace name context forever, level 12:10270 trace name context
forever, level 10';
High parse ratio.

It is important to identify what could produce high parsing calls:
-
Use of dynamic plsql

Execution of DDL statements during periods of high workload. Every time a DDL statement
is executed, it will cause invalidation of all the statements referencing the object involved.
Next time a sql statement referencing the object is executed, it will have to be reparsed and
loaded into the shared pool.
Typical operations that cause this situation is the execution of:
-
Grant/revoke command
Alter view
Alter package | procedure
Analyze table |index
Dbms_stats
Truncate table
Alter index
Alter table move
Direct effects of this situation will be an increase in the parsing calls but also will be a latch
contention for the library cache latch. If the invalidation/reloads cause fragmentation of the
shared pool, error ora-4031 will be reported.
The diagnostic
View v$librarycache is the first reference, particular the reloads and invalidation columns. This view
is used by statspack and the AWR repository in 10g. Also dumping the librarycache at level 1 will
dump output of this view.
How to identify if are DDL statements executed?

Dumping the library cache at level 11 and it needs to be close to the period of time where high
reloads/invalidations is observed.
SQL> oradebug setmypid
SQL> oradebug dump library_cache 11
15/40

- 16 SQL> oradebug tracefile_name
Or
SQL> alter system set events immediate trace name library_cache level 11;
The trace file created will have to be browsed and identify calls to DDL statements. It is
important to mention that dumping a library cache will hold the library cache latch and if the
system is suffering for contention on this latch, then more contention will be added. Also the size
of the shared pool makes a difference, a large shared pool size will require more time and more
space for the trace file.
Using view x$ksmlru

This is view is used to determine the object(s) causing more reloads into the library cache.
Select KSMLRCOM, KSMLRHON, KSMLRNUM from x$ksmlru where KSMLRSIZ > 5000;
COLUMN
KSMLRCOM
KSMLRSIZ
KSMLRNUM
KSMLRHON
DESCRIPTION
allocation comment that describes the type of allocation.
amount of contiguous memory being allocated. Values over around 5K
start to be a problem, values over 10K are a serious problem, and values over
20K are very serious problems. Anything less then 5K should not be a problem
number of objects that were flushed from the shared pool in order
allocate the memory
the name of the object being loaded into the shared pool if the
object is a PL/SQL object or a cursor.
This query will show the name of the object being loaded into the shared pool (KSMLRHON)
and how many objects were flushed out from the shared pool.
What to Do?
Objects causing a large number of other objects been flushed out from the shared pool are candidates
to be pinned into the shared pool using dbms_shared_pool.keep procedure.
The content of the view is recycled every time the object is consulted, either spool the content to a
file or store in a temporary table.
In the statspack reports or AWR reloads/invalidations can be found in the library cache section.
16/40

- 17 Library Cache Activity DB/Inst: ATHR/ATHR7 Snaps: 321-322
-> "Pct Misses" should be very low
Get
Pct
Pin
Pct
InvaliNamespace
Requests
Miss
Requests
Miss
Reloads dations
--------------- ------------ ------ -------------- ------ ---------- -------BODY
48,587
0.3
79,434
0.2
15
0
CLUSTER
28,873
0.0
30,343
0.1
10
0
INDEX
11,375
56.5
12,886
56.7
286
0
SQL AREA
2,045,362
1.4
6,466,687
0.7
5,936
598
TABLE/PROCEDURE
188,394
6.5
568,846
5.8
4,533
0
TRIGGER
67,852
0.1
68,005
0.1
1
0
-------------------------------------------------------------
17/40

- 18 -
DATABASE ISSUES CAUSING ERRORS ORA-4031

It has been mentioned that error ora-4031 indicates lack of space in the shared pool that can be caused by
a real lack of space or due to leaks of space caused by Oracle bugs.
a. Large pool not been used.
Initially the shared pool is used by all the sqls executed in the database including plsql and all the
structures required, like state objects, library cache objects, etc. In addition there are other components
like MTS , parallel query process and IO slaves processes that could use the shared pool when the large
pool is not configured.
Before 10g space from large pool is automatically used if parameter parallel_automatic_tuning (PAT) is
set to TRUE or if PAT was set to FALSE, setting _ px_large_pool to TRUE force the allocation of this
space.
If large pool was not configured, MTS, parallel query or IO slaves will use space from the shared pool
area. In 10g, parallel_automatic_tunning is not required, as soon as large_pool_size is configured, the
pool automatically will be used.
A good tuned database using MTS, parallel query or IO slaves will require the configuration of
large_pool_size parameter.
Output from v$sgastat will show if large pool is allocated. Allocations for PX msg pool will be under
shared pool if large pool is not configured.
Case study 4 describes this type of problem.
b. Incorrect size for shared pool.
An estimate of the size for shared pool can be configured following Note 1012046.6: HOW TO
CALCULATE YOUR SHARED POOL SIZE.
The queries referenced in the note should be executed once the shared pool has been fully used in order
to obtain a precise image
A common mistake executed by many users is running a new version of the database with the
configuration used in the previous version. First, there are architectural changes that will consume more
memory. An example of this is the allocation session parameters. Figure 4.1
18/40

- 19 SQL> select 'sga heap('||KSMCHIDX||',0)' sga_heap,ksmchcom contents,
count(*) chunks,sum(ksmchsiz) "total bytes",avg(ksmchsiz) "avg bytes"
from sys.x$ksmsp where inst_id = userenv('Instance')
and ksmchcom like 'session%'
group by 'sga heap('||KSMCHIDX||',0)' ,ksmchcom
SGA_HEAP
CONTENTS
CHUNKS Total Bbytes Avg Bytes
sga heap(1,0)
session parame
25
589700
23588
Figure 4.1. Displaying memory allocation for a particular element (comment).

In 10g the average size was 23k and is normally allocated for each session. This value was lower in 9i,
8i and 7.
There is not a formula to use and calculate the new size for the new release. The available tools are the
guidelines from the documentation during the upgrade and Note 1012046.6.
How can I know if the system is running out of memory but no error ora-4031 has been reported?
The shared pool has a similar mechanism to the LRU that helps to reuse the freeable space, avoiding
errors ora-4031. A consequence of this will be an increment in reloads which could cause fragmentation
or contention in the library cache. By the end is necessary to determine how much free memory is
available.
Normally view v$sgastat is used to track free memory but this view reports a global number, including
the total value through all subpools and reserved memory. This information has to be detailed by
subpools because free memory could be available in one specific subpool or could be available only in
the reserved area which only is used based on parameter _shared_pool_reserved_min_alloc. The default
value is 4400 bytes and memory will be allocated from this area only if there is not space in the normal
shared pool and the space requested is larger than _shared_pool_reserved_min_alloc.
19/40

- 20 To monitor the free memory use query listed in table 4.1.
col sga_heap format a15
col size format a10
select 'sga heap('||KSMCHIDX||',0)'sga_heap,ksmchcom ChunkComment,
decode(round(ksmchsiz/1000),0,'0-1K', 1,'1-2K', 2,'2-3K',
3,'3-4K',4,'4-5K',5,'5-6k',6,'6-7k',7,'7-8k',8,'8-9k', 9,'9-10k','> 10K') Size, count(*),
ksmchcls Status, sum(ksmchsiz) Bytes
from x$ksmsp
where KSMCHCOM = 'free memory'
group by 'sga heap('||KSMCHIDX||',0)',ksmchcom, ksmchcls,
decode(round(ksmchsiz/1000),0,'0-1K', 1,'1-2K', 2,'2-3K',
3,'3-4K',4,'4-5K',5,'5-6k',6,'6-7k',7,'7-8k',8,'8-9k', 9,'9-10k','> 10K')
SGA_HEAP
CHUNKCOMMENT Size
COUNT(*) STATUS
BYTES
sga heap(1,0)
sga heap(1,0)
sga heap(1,0)
free memory
free memory
free memory
1055
1
15
54012
2300
3193500
0-1K
2-3K
>10K
free
free
R-free
Table 4.1. Query that displays how free memory is distributed.

Table 4.1, shows that most of the free memory is available under the range of 0-1K and the average size
is very small. A system with a similar output is running with very few free memory and will be exposed
to errors ora-4031.
Another option used to review the free memory is dumping a heapdump level 2 (shared pool) and
running tool heap.awk. This tool redirects the output to a file and the free memory distribution can be
found looking for Free List Bucket Summary:
Bucket 0 [size=32
] Count= 0 Av.Size=
0.00 Max=
Bucket 1 [size=40
] Count= 443 Av.Size= 40.00 Max=
Bucket 2 [size=48
Bucket 3 [size=56
Bucket 4 [size=64
Bucket 5 [size=72
Bucket 6 [size=80
0
40
48
56
64
72
80
Every bucket shows the number of chunks, the average size and the maximum chunk size. Bucket 1
indicates stores chunks between 0 and 48 bytes and there are 443 of these chunks. On the other side,
buckets for large size of memory will have 0 chunks. A system running out of free memory will show a
similar report.
20/40

- 21 c. Multiple subpools
9iR2 introduced multiple subpools for the shared pool looking to reduce contention in the library cache.
The subpools are automatically configured based on the size of shared pool and number of cpus
available.
Configuring a shared pool of 1gb with four (4) subpools, each one will have 256mb of available memory.
A process is attached to a particular subpool and can not share space from other subpools. If the subpool
runs out of space, error ora-4031 will be reported.
Primarly bugs on this area were caused due to an unbalanced allocation of space. In other words, the
process allocates space primarly from a particular subpool. Even the others subpools had free space,
after running out of space, error ora-4031 was raised. Unfortunately there is not a reallocation of the
process to a different subpool.
How to identify if the sytem is having multiple subpools?
Running the following query:
select 'sga heap('||KSMCHIDX||',0)'sga_heap
from x$ksmsp
group by 'sga heap('||KSMCHIDX||',0)'
Another form is reviewing the trace file created by heapdump level 2 (shared pool). When tool
heap.awk is used, the resultant file will present the information for every subpool, using the tittle :
---> HEAP DUMP heap name="sga heap(6,0)" .
The same trace file created by heapdump event can be read through grep command like this:
grep sga heap( trace_file.trc
How to workaround error ora-4031 due to multiple subpools?
Running with multiple subpools reduce contention on the library cache latch, but also every subpool will
have minor amount of memory available. The option is using only one pool like in Oracle 8i and it is
controled through parameter _kghdsidx_count = 1.
Systems with a large number of cpus will create up to 7 or 8 subpools. If contention in the latch library
cache is an issue, this parameter can be adjusted to have 2 or 3 subpools.
A typical scenario of this error is when the system is upgraded from Oracle8i or Oracle90 to Oracle9iR2
using same size for shared pool. There is not a mechanism or formula that indicates how much extra
memory should be allocated or how many subpools should be defined.
21/40

- 22 When finding errors ora-4031 and having multiple subpools, first step will be using one large subpool
(parameter _kghdsidx_count = 1). If library cache contention is an issue, this parameter could be
adjusted to have more subpools, increasing the memory.
d. Overallocation of permanent memory row cache RAC GES
e. Memory not been reused large freeable allocations
Parameters ruling shared pool space

Shared_pool_size
Large_pool_size
Shared_pool_reserved_size
Parameters affecting shared pool manageability
_kghdsidx_count
_shared_pool_reserved_min_alloc
sga_target
Cursor_space_for_time
Session_cache_cursors
Cursor_sharing
22/40

- 23 -
METHODOLOGY FOR THE DIAGNOSTIC

There is not a standard methodology to work with errors ora-4031, but below is a general procedure that
will help to collect information and start with the initial diagnostic.
In 9206 was introduced an enhancement that dumps a trace file when error ora-4031 is produced. This
trace file can be used to easily identify if the shared pool is distributed in multiple subpools, which area is
allocating more memory, how is the permanent memory allocated.
1. Generate heapdump
In addition to any trace file generated by the 4031 diagnostic, obtain a complete heapdump:
SQL>oradebug setmypid
SQL>oradebug unlimit
SQL>oradebug dump heapdump 536870914
Level 536870914 will dump the SHARED POOL heap plus 5 largest subheaps and 5 largest subheaps for
each one.
2. Use heap.awk
awk f heap1_4.awk <tracefile_name> > <output_file_name>
The file create by heap.awk will provide the initial information for the first diagnostic. Notice that error
ora-4031 is not generated by default in the alert.log. If it is required, enable event errorstack for error
ora-4031:
alter system set events 4031 trace name errorstack level 10
To turn it off:
alter system set events 4031 trace name context off
The case studies included in this document will explain with more detail the procedure applied to each
case. It will help to get a better understanding of the common areas that are present on the heap dump,
common areas present on the error, how to search in webiv or the source code.
23/40

- 24 -
CASE STUDIES
CASE 1. 4358485.996
PROBLEM
Testing the upgrade to 9i of a pre-production database . Hundreds of errors ora-4031
Shared pool size = 500M
Multiple subpools (2)
After first errors ora-4031 shared pool was increased from 250m to 400m and finally to 500m
DIAGNOSTIC
There was not available a trace file created by error ora-4031. A heapdump level 2 was requested during
a period of high workload in order to analyze the distribution of space in the shared pool.
Running tool heap.awk using the trace file created by heapdump produce the following information:
---> HEAP DUMP heap name="sga heap(1,0)" desc=38002bd20
Type
Count
Sum
Average
-----------------------------------------------------------R-freeable
76
1003536
13204.42
R-perm
44
3258720
74061.82
R-free
132
57088752
432490.55
perm
36868 176831808
4796.35
recreate
3935 2491256
633.10
R-recreate
2
1368
684.00
---> HEAP DUMP heap name="sga heap(2,0)" desc=380034888
Type
Count
Sum
Average
------------------------------------------------------------------R-freeable
84
1134544
13506.48
R-perm
53
2483864
46865.36
R-free
150
54004784
360031.89
perm
36633 147482896
4025.96
freeable
29376 31449192
1070.57
recreate
17902 12335184
689.04
free
14594 17005384
1165.23
Notes:
24/40

- 25 1. The shared pool size is around 500mb but it is distributed in two subpools of 256m (aprox) each.
Both subpools show an overallocation of permanent memory larger than 31%.
Section Two. Breakdown section
Because most of memory was allocated as permanent and there were not comments stored in the shared
pool, these section did not provide information.
Section Three. Free List Bucket Summary
Free List Bucket Summary
Bucket 0 [size=32
] Count= 0 Av.Size=
0.00 Max=
0
Bucket 1 [size=40
40
Bucket 2 [size=48
48
/* DATA CUT TO SAVE SPACE */
Bucket 232 [size=3208
3248
3296
] Count= 0 Av.Size=
0.00 Max=
0
Each subpool presented the same picture. Buckets on top of the list (small size) have space but buckets
on the bottom (large size) had 0 chunks available.
Notes:
1. The free memory available in each subheap wain small chunks, where the largest chunk was 3k . Error
ora-4031 were reported because the requested size was 4096 (from the alert.log file)
ORA-04031 : unable to allocate 4096 bytes of shared memory ("shared pool","DECLARE
v_hostname VARCHAR...","PL/SQL MPCODE","BAMIMA: Bam Buffer")
Reviewing the history of errors ora-4031 (file alert_prem.log.051905), running a grep '04031'
alert_prem.log.051905, we can observe that all errors were requesting less than 4200 bytes.
This error could be aliviated modifiying parameter _shared_pool_reserved_min_alloc. By default this
parameter is set to 4400. How this works is:
if !(request_from_shared_pool(size)) then
if size >" _shared_pool_reserved_min_alloc" then
request_from_reserverd_shared_pool
fi
fi
If there is not space in the shared pool area (defined by parameter shared_pool_size) only if the space
requested is larger than parameter _shared_pool_reserved_min_alloc, then Oracle tries to allocate from
25/40

- 26 the reserverd shared pool.
Setting _shared_pool_reserved_min_alloc to 4000 will use the memory from the reserved pool, but this is
only a workaround.
ACTION PLAN
1. This machine has 8 cpus and that is why automatically Oracle is distributing the shared pool in 2
subpools. As the shared pool size is not big, it is recommended to have only one pool. Please adjust
parameter _kghdsidx_count = 1 and restart the instance
2. Increase shared pool size or change parameter _shared_pool_reserved_min_alloc to 4000.
3. In order to investigate the largest perm allocations, it will be required to set the following event:
alter system set events '10235 trace name context forever, level 65536' or at the parameter file
event="10235 trace name context forever, level 65536"
26/40

- 27 CASE 2. 4413750.999
PROBLEM
A production database migrated from 9i to 10gR1. The closing of the month is the busiest period of the
system and error ora-4031 was reported.
Original configuration of shared pool was 500m. and was increased to 1000m after first occurance of
error ora-4031. Parameter cursor_sharing set to SIMILAR (since 9i).
DBA discovered sql area in v$sgastat is consuming more then 600mb.
DIAGNOSTIC
The first response to error ora-4031 was to increase the shared pool and error ora-4031 was not
reproduced. The DBAs identify a large allocation in the sql area ( from v$sgastat).
If the space in the shared pool is growing it is because shared cursors are not shared. The following
select was executed:
select sa.sql_text,sa.version_count ,ss.* from v$sqlarea sa,v$sql_shared_cursor ss

where sa.address=ss.address
and sa.version_count > 500
order by sa.version_count desc;
There was around 2900 records returned by the query, indicating it was a lot of sqls not been shared.
SQL_TEXT
--------------------------------------------------------------------------------------------------VERSION_COUNT SQL_ID
ADDRESS
CHILD_ADDRESS
CHILD_NUMBER U S O
O S L S E B P I S T
------------- ------------- ---------------- ---------------- ------------ - - - - - - - - - - - A B D L T R I I R L I O S M U T N F A I T D L D B P
- - - - - - - - - - - - - - - - - - - - - - - - - SELECT REO_DISBRMT_LNE_ITM.REO_DISBRMT_LNE_ITM_OID,
REO_DISBRMT_LNE_ITM.REO_DISBRMT_OID, REO_DISBRMT
_LNE_ITM.PD_AMT, REO_DISBRMT_LNE_ITM.REO_DISBRMT_LN_ITM_TYP_CD,
REO_DISBRMT_LNE_ITM.RQSD_AMT, REO_DI
SBRMT_LNE_ITM.AUD_VER, REO_DISBRMT_LNE_ITM.LAST_UPD_DT FROM REO_DISBRMT_LNE_ITM
REO_DISBRMT_LNE_ITM
WHERE REO_DISBRMT_LNE_ITM.REO_DISBRMT_OID IN
27/40

- 28 1015 bf4vj4w7uzx34 000000043EF39FB0 000000042FC21EA8
N N N N N N N N N N
N N N N N N N N N N N N N N N N N N N N N N N N N N
14 N N N
Normally with cursor sharing problems view v$sql_shared_cursor is used and the cause of the problem
could be identified if columns after child_number column show value Y. Unfortunately in this case for
all the 2900 records returned all columns showed N.
The next step was identifying for a particular sql the application where it was called and set events 10046
and 10270.
oradebug setorapid <oracle process id>
oradebug event 10046 trace name context forever, level 12:
10270 trace name context forever, level 10
oradebug tracefile_name
The trace file created after setting the event showed that when using literals with a different column
datatype, the type convertion marked the bind variable as unsafe and that cause the generation of a child
cursor.
The cursor used was :
SELECT REO_DISBRMT_LNE_ITM.REO_DISBRMT_LNE_ITM_OID,
REO_DISBRMT_LNE_ITM.REO_DISBRMT_OID,
REO_DISBRMT_LNE_ITM.PD_AMT,
REO_DISBRMT_LNE_ITM.REO_DISBRMT_LN_ITM_TYP_CD,
REO_DISBRMT_LNE_ITM.RQSD_AMT
, REO_DISBRMT_LNE_ITM.AUD_VER, REO_DISBRMT_LNE_ITM.LAST_UPD_DT
FROM CLM.REO_DISBRMT_LNE_ITM
WHERE REO_DISBRMT_LNE_ITM.REO_DISBRMT_OID IN
END OF STMT
The bind variable was marked unsafe. See oacfl2=0x300. In this example dty=1 is NUMBER but check
the value between which is a literal.
BINDS #1:
bind 0: dty=1 mxl=32(01) mal=00 scl=00 pre=00 oacflg=10 oacfl2=0300 size=32 offset=0
bfp=ffffffff7bc7a838 bln=32 avl=01 flg=09
value="1"
28/40

- 29 CONCLUSION
Based on this diagnostic a testcase was created and the problem was reproduced. Tested in Oracle9i
showed different results (no child cursors).
As a workaround it was suggested to change cursor_sharing to FORCE which also did not produce child
cursors. It was not accepted by customer as it could change the execution plan in other processes.
A BUG 3844015 was opened and the root of the problem is still under investigation but tied to some
changes caused by other bugs.
29/40

- 30 CASE 3. TAR 4383850.996
PROBLEM
Two nodes RAC running 9204 with PeopleSoft software. Shared pool size 500mb in each node.
The error was intermittent and the only workaround was reclying the instances.
DIAGNOSTIC
There was not a pattern in all times the error was reproduced. A heapdump level 2 was generated by
LMON process, which makes the cause of the problem RAC related.
Here is the description of every section in the output file after running heap.awk
---> HEAP DUMP heap name="sga heap(1,0)" desc=3800294d0
Type
Count
Sum Average
---------------------------------------------------------------R-freeable
70
2800
40.00
R-free
35
29527120
843632.00
perm
57918 504515152
8710.85
freeable
1171 3010016
2570.47
recreate
2312 1572224
680.03
free
37431 48587432
1298.05
All the memory was allocated as PERMANENT and was almost 80% of the shared pool. Because perm
memory does not store comments, it was not possible at this time to see the breakdown. The next action
was to enable event 10235 level 65536 and dump heapdump level 2 once the shared pool reaches the
regular usage.
LMON process failed with error ora-4031 trying to allocate memory for the GES enqueues area.
ORA-04031: unable to allocate 2840 bytes of shared memory ("shared pool","unknown object","sga
heap(1,0)","ges enqueues")
The permanent allocations was not available in the heapdump but another source of information was the
output from v$sgastat executed one the shared pool was busy again, where it displayed large allocations
for the GES resources:
30/40
RUNTIME
6/27/2005 1:46:49 PM
6/27/2005 1:46:49 PM
6/27/2005 1:46:49 PM
6/27/2005 1:46:49 PM
NODE
PS88PRD1
PS88PRD1
PS88PRD1
PS88PRD1
POOL
shared pool
shared pool
shared pool
shared pool

- 31 NAME
BYTES
ges enqueues
127463368
ges resources
189889120
ges big msg buffers
1110888
ges reserved msg buffers
628808
CONCLUSION
Running a search in WebIV (words 4031 ges enqueues rac) provide the following bug:
3046725 ORA-4031 - SHARED_POOL FRAGMENTED WITH HIGH GES RESOURCE
& ENQUEUES
2563301 ORA-4031 possible in RAC environment under load
Customer implemented workaround of bug 2563301 using parameter _lm_cache_res_cleanout=70 to
increse the amount of cache resource to be clean out in each attempt.
This problem was due to the initial implementation for the GES resource and GES enqueues allocations
(global resources) in RAC environments, which are allocated as permanent allocations. Permanent
allocations are only released when the instance is recycled.
The cached resource state objects could be built up overtime. It could consume a lot of shared
pool memory. Since state objects could not be freed back to shared pool once they are allocated, it
may use up more and more shared pool memory if the cached resources were not cleanup fast
enough.
31/40

- 32 CASE 4. TAR 4417034.995
PROBLEM
Database running in a Sun 10K with 16 CPUs and a shared pool size of 3.3gb, RDBMS 9206. Process
generating error ora-4031 was dbwr and as a consequence instance crashed.
DIAGNOSTIC
There were two trace files created by the new enhancement introduced in 9205 and the trace files
generated at different times were used to find a common pattern.
The trace file created by this improved diagnostic has a section where indicates what type of allocation
request failed and caused error ora-4031 and the subpool (if multiple subpools configured). The
allocation request does not indicate is the culprit of error ora-4031 and only could be the victim.
Allocation Request Summary Information
======================================
Current information setting: 00654fff
Dump Interval=300 seconds SGA Heap Dump Interval=3600 seconds
Last Dump Time=05/02/2005 18:45:34
Allocation request for: ksfv messages
Heap: 3800562b8, size: 2464
******************************************************
HEAP DUMP heap name="sga heap(6,0)" desc=3800562b8
extent sz=0xfe0 alt=200 het=32767 rec=9 flg=-126 opc=0
parent=0 owner=0 nex=0 xsz=0x1
Figure 6.1. Information from trace file created when error 4031 is produced. This
improvement is included since 9205.
From Figure 6.1 we identify the type of allocation request KSFV MESSAGES and the subpool SGA
HEAP (6,0).
Both failures of dbwr process was for same type of request and searching in WebIV help to identify that
this allocation (KSFV MESSAGES) is associated with IO SLAVES.
Reviewing the instance configuration showed parameter dbwr_io_slaves was enabled (4) and parameter
large_pool_size was not enabled.
The IO SLAVE processes should be assigned to the large pool but if it is not configured the space is
alloacated in the shared pool size.
After configuring parameter large_pool_size the problem was not reproduce.
There is one bug active related to the allocation of memory for the IO SLAVES bug 3291723
32/40

- 33 CASE 5. TAR 4326100.996
PROBLEM
A database migrated from 817 to 9205 using a shared pool size of 738197504 bytes. Instance failed with
errors ora-4031.
DIAGNOSTIC
The information in the trace file created by heapdump level 2 showed the shared pool was divided in
four (4) subpools, 184549376 bytes each.
The file created by heap.awk showed the following information:
Section 1. General Space Allocation
When the shared pool is distributed in subpools it is important to review the information for each
subpool. To save space in the document we are going to include information for one subpool.
All four (4) subpools presented a similar situation where the largest allocations of memory were of type
perm and freeable. Remember that if permanent allocations is larger than 31% of shared pool, it has to
be reviewed.
---> HEAP DUMP heap name="sga heap(1,0)" desc=5802b1a0

Type
Count
Sum
Average
--------------------------------------------------------------R-freeable
270
6110360
22630.96
R-perm
82
7944072
96878.93
R-free
112
1129216
10082.29
perm
9486 114291672
12048.46
recreate
23403 16845000
719.78
freeable
81882 135875936
1659.41
R-recreate
2
3168
1584.00
free
12184 19797680
1624.89
There was also a particular situation where every subpool was aproximately around 180mb but the
memory consumed was larger. We were not able to identify why but the investigation was centered on
identifying the high usage of permanent allocations. Due to lack of comments for permanent allocations,
event 10235 level 65536 was required.
33/40

- 34 Section Two. Breakdown section
Section one showed around 19m of free memory in subpool one with an average chunk of 1624 bytes.
Other subpools showed similar numbers. That indicates the free memory was limited to smaller chunks
and allocations larger than 27k will report error ora-4031.
]
]
]
]
]
]
]
]
Count= 47 Av.Size=
Count= 56 Av.Size=
Count= 80 Av.Size=
Count= 36 Av.Size=
Count= 53 Av.Size=
Count= 279 Av.Size=
Count= 1103 Av.Size=
Count= 706 Av.Size=
3853 Max=
3905 Max=
3950 Max=
4004 Max=
4068 Max=
4586 Max=
10652 Max=
20164 Max=
3872
3920
3968
4016
4112
8120
16384
27456
Section Four. Breakdown of CPRM Chunks (Commented Perm Chunks)

When event 10235 level 65536 is enabled and the heapdump is executed, the trace file will have a section
for all chunks associated to this type of memory. The section starts with title PERMANENT CHUNKS.
Every permanent chunk will have the following format:
4eb38e5b8 sz=
16400
cprm
"trace buffer "
where
4eb38e5b8 is the memory address in the shared pool
sz= 16400 is the amount of permanent memory allocated
cprm trace buffer is the comment
PERMANENT CHUNKS:
Chunk
4eb38e598 sz= 20616

4eb38e5b8 sz= 16400
4eb3925c8 sz=
56
4eb392600 sz= 4112
perm
cprm
cprm
cprm
"perm
" alo=20616
"trace buffer "
"kgl lock hash t"
"kgllk hash tabl"
Now tool heap.awk will create a summary for the permanent chunks, which in this document has been
defined as section four: Breakdown of CPRM Chunks (Commented Perm Chunks)
34/40

- 35 Type
Count
Sum
Average
-------------------------------------------------------------------enqueue resourc
656
1759288
2681.84
library cache
20527
86814312
4229.27
kzulu
538
38736
72.00
call
495
1547040
3125.33
trace buffer
3460
56744000
16400.00
kgl lock hash t
3453
193368
56.00
kgllk hash tabl
3453
14198736
4112.00
trace buf hdr x
108
3984768
36896.00
obj stat memor
90
384480
4272.00
object level s
39
165360
4240.00
kglsim heap
2808
11726208
4176.00
kglsim object b
5498
22255904
4048.00
modification h
4
17088
4272.00
state objects
10
31544
3154.40
Sort Segment e
17
72624
4272.00
monitoring col
9
38304
4256.00
constraints
1
1646056
1646056.00
event statistic
6606
74504416
11278.29
Figure 6.2. Output from heap.awk using a trace file created by heapdump level 2
and event 10235 level 65536.
Figure 6.2 shows that the largest allocations of permanent memory comes from:
library cache
trace buffer
kgllk hash tabl
kglsim heap
kglsim object b
event statistic
20527
3460
3453
2808
5498
6606
86814312
56744000
14198736
11726208
22255904
74504416
4229.27
16400.00
4112.00
4176.00
4048.00
11278.29
For the library cache allocations there is not much to do as they are required for the normal functioning
of the database. Kglsim allocations are related with the library cache simulators, introduced in 9iR2.
There is around 33mb. Trace buffer is part of the new tracing facility and is consuming around 50mb.
These features can be disabled with the following parameters:
Statistic_level=basic. At this value will turn off all the simulators. _library_cache_advice = FALSE will
turn off library cache simulator only.
Trace_enabled=false. Turn off trace generation
35/40

- 36 -
APPENDIX A - Important SQLs

The most important tool to work on 4031 issues is the heapdump event together with the heap.awk tool.
The process of generating the trace file and loading it to metalink , downloading to a local machine and
runing the heap.awk is time consuming.
Below there is a reference to some queries that will report information in same format produced by
heap.awk tool.
Free memory distribution
This query will show in which range the free memory is distributed, initially in ranges of one (1)
kilobyte. This query can be used to identify if the shared pool has sufficient memory (large value for
the range > 10K) or if there is a small amount of memory (values displayed on the small ranges, like
1 2k.). The query includes the information for every subpool and also the free memory from the
reserved pool.
col SubPool format a15
select 'sga heap('||KSMCHIDX||',0)' Subpool,ksmchcom ChunkComment,
decode(round(ksmchsiz/1000),0,'0-1K', 1,'1-2K', 2,'2-3K',3,
'3-4K',4,'4-5K',5,'5-6k',6,'6-7k',7,'7-8k',8,'8-9k', 9,'9-10k','> 10K') Size Range,
count(*) Count,
ksmchcls Class , sum(ksmchsiz) Bytes
from x$ksmsp
where KSMCHCOM = 'free memory'
group by 'sga heap('||KSMCHIDX||',0)',ksmchcom, ksmchcls,
decode(round(ksmchsiz/1000),0,'0-1K', 1,'1-2K', 2,'2-3K', 3,
'3-4K',4,'4-5K',5,'5-6k',6,'6-7k',7,'7-8k',8,'8-9k', 9,'9-10k','> 10K')
order by ksmchcls;
Subpool
CHUNK
COMMENT Size
Count
Class
BYTES
-------------------------------------------------------------------------------------sga heap(1,0) free memory
> 10K
2
R-free
335720
sga heap(2,0) free memory
> 10K
1
R-free
167860
> 10K
1
free
16110576
0-1K
156
free
8524
2-3K
1
free
1888
> 10K
1
free
2000020
36/40

- 37
Free Memory distribution by buckets.

This query reports the distribution of free memory for each bucket as it is showed in section three
on the report created by heap.awk tool.
select KSMCHIDX "SubPool",
decode(sign(ksmchsiz - 812),-1, (ksmchsiz - 16) / 4,
decode(sign(ksmchsiz - 4012),-1, trunc((ksmchsiz + 11924) / 64),
decode(sign(ksmchsiz - 65548),-1, trunc(1/log(ksmchsiz - 11, 2)) + 238,254)
)
) bucket, sum(ksmchsiz) free_space,
count(*) "Count", trunc(avg(ksmchsiz)) "Avg. Size", max(ksmchsiz) "Max. Size"
from sys.x$ksmsp
where inst_id = userenv('Instance') and ksmchcls = 'free'
group by KSMCHIDX, decode(sign(ksmchsiz - 812),-1, (ksmchsiz - 16) / 4,
)
)
order by KSMCHIDX, decode(sign(ksmchsiz - 812),-1, (ksmchsiz - 16) / 4,
)
)
General Space Allocation

This query will show how the memory is allocated in the shared pool, including the reserved pool
and including each type (permanent, freeable or recreatable). This is the same information from
section one found in the report created by heap.awk tool.
Col SubPool format a15
Set pagesize 100
SELECT 'sga heap('||KSMCHIDX||',0)' "SubPool" ,KSMCHCLS CLASS,
COUNT(KSMCHCLS) Count, SUM(KSMCHSIZ) Sum,avg(KSMCHSIZ) Average
FROM X$KSMSP GROUP BY 'sga heap('||KSMCHIDX||',0)',KSMCHCLS;
SubPool
CLASS
COUNT
SUM AVERAGE
---------------------------------------------------------------------------------sga heap(1,0) free
1
16110576
16110576
sga heap(1,0) perm
1
16609280
16609280
sga heap(1,0) recr
369
233776
633.539295
sga heap(1,0) R-free
2
335720
167860
sga heap(1,0) R-freea
4
80
20
sga heap(1,0) freeabl
25
264928
10597.12
sga heap(2,0) free
159
1812892
11401.8365
37/40
sga heap(2,0)
sga heap(2,0)
sga heap(2,0)
sga heap(2,0)
sga heap(2,0)
perm
recr
R-free
R-freea
freeabl
1
1
2
2600

- 38 10456556
10456556
2177 1416456
650.645843
167860
167860
40
20
2923376
1124.37538
Breakdown
The following query displays the memory allocated but this time using the comments stored in
the shared pool. This is same information displayed in section two on the report created by
heap.awk tool.
col SubPool format a15
select 'sga heap('||KSMCHIDX||',0)' "SubPool",ksmchcom contents, count(*)
chunks,sum(ksmchsiz) "Tot.Bytes",avg(ksmchsiz) "Avg.Bytes"
from sys.x$ksmsp where inst_id = userenv('Instance')
--and ksmchcls not like 'R%'
group by 'sga heap('||KSMCHIDX||',0)' ,ksmchcom
SubPool
CONTENTS
CHUNKS Tot.Bytes Avg.Bytes
-------------------------------------------------------------------------------------sga heap(1,0) KQR PO
347
177396
511.227666
sga heap(1,0) KGK heap
1
3268
3268
sga heap(1,0) joxs heap
1
92
92
sga heap(1,0) PX subheap
1
52472
52472
3
16446296
5482098.67
sga heap(1,0) KGK contexts
2
2376
1188
sga heap(1,0) MS alert log
1
10252
10252
sga heap(2,0) KQR PO
413
178500
432.20339
sga heap(2,0) errors
12
10512
876
sga heap(2,0) KGK heap
1
560
60
sga heap(2,0) sql area
694
1527912
2201.60231
sga heap(2,0) KGLS heap
475 645916
1359.82316
sga heap(2,0) parameters
1
1072
1072
sga heap(2,0) KGL handles
743 268408
361.248991
163
1952768
11980.1718
sga heap(2,0) PL/SQL DIANA
237 1453976
1915.51055
38/40

- 39
Monitoring specific allocation.

The following script is very helpful when monitoring a leak for a specific allocation of memory.
It is more appropiate than generating heapdumps.
select ksmchidx,ksmchcom,count(*) ,sum(ksmchsiz)
from x$ksmsp where
ksmchcom like 'KQR%'
group by ksmchidx,ksmchcom;
KSMCHIDX KSMCHCOM
COUNT(*) SUM(KSMCHSIZ)
---------------------------------------------------------------------------------1
KQR PO
347
177396
2
KQR PO
413
178500
Large Pool Monitoring

select ksmdsidx "SubPool",ksmssnam "Type" ,ksmsslen "Bytes"
from x$ksmls where ksmsslen != 0;
SubPool Type
Bytes
----------------------------------------------------------1 free memory
33522208
1 PX msg pool
32224
2 free memory
33522208
2 PX msg pool
32224
If shared pool is configured with multiple subpools, large pool will also have same configuration.
Changing parameter _kghdsidx_count will affect large and shared pool.
39/40

- 40 APPENDIX B. List of trace files
This is the trace files and output files created for each case study.
CASE 1
CASE 2
CASE 3
CASE 4
CASE 5
4358485.996
4413750.999
4383850.996
4417034.995
4326100.996
rem_ora_23985.trc
ps88prd2_lmd0_2115.trc
p20_dbw0_16403.trc
ssbp_ora_83509.trc
40/40

Resolve ORA-4031 Error in 4 Steps

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Resolve ORA-4031 Error in 4 Steps

Hochgeladen von

Copyright:

Verfügbare Formate

How to resolve ORA-4031 error

Jaime Figueroa - Advanced Resolution

APPENDIX A - Important SQLs.....................................................................................................2

How to resolve ORA-4031 error

How to resolve ORA-4031 error

SHARED POOL OVERVIEW

Free List Buckets

Every bucket will have a free list

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

TOOLS AVAILABLE TO COLLECT INFORMATION

How to resolve ORA-4031 error

6216 R-freeable "library cache " ds=3c6ce6130

How to resolve ORA-4031 error

How to resolve ORA-4031 error

Section One. General Space Allocation

---> HEAP DUMP heap name="sga heap(1,0)" desc=38002a9b0

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

ORA-4031 CAUSED FROM THE APPLICATION SIDED

Application not using bind variables.

How to resolve ORA-4031 error

Multiple child cursors

How to resolve ORA-4031 error

The child cursor is an explain plan cursor and should not

Insufficient privileges on objects referenced by the

How to resolve ORA-4031 error

High parse ratio.

Use of dynamic plsql

How to identify if are DDL statements executed?

How to resolve ORA-4031 error

Using view x$ksmlru

How to resolve ORA-4031 error

How to resolve ORA-4031 error

DATABASE ISSUES CAUSING ERRORS ORA-4031

How to resolve ORA-4031 error

Figure 4.1. Displaying memory allocation for a particular element (comment).

How to resolve ORA-4031 error

Table 4.1. Query that displays how free memory is distributed.

How to resolve ORA-4031 error

How to resolve ORA-4031 error

Parameters ruling shared pool space

How to resolve ORA-4031 error

METHODOLOGY FOR THE DIAGNOSTIC

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

select sa.sql_text,sa.version_count ,ss.* from v$sqlarea sa,v$sql_shared_cursor ss

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

How to resolve ORA-4031 error

---> HEAP DUMP heap name="sga heap(1,0)" desc=5802b1a0

How to resolve ORA-4031 error

Section Four. Breakdown of CPRM Chunks (Commented Perm Chunks)

"trace buffer "

4eb38e598 sz= 20616

How to resolve ORA-4031 error