OpenEdge Release 10.

1B and higher
RDBMS Shared Memory

February 13, 2008

By Keith Sanborn and Gus Bjorklund

Overview

A set of shared memory enhancements were introduced in OpenEdge Release 10.1B that enable
increased shared memory addressability and better management of the shared memory allocation
process. Increased shared memory addressability allows larger pools of shared memory for
databases, which can reduce operating system resource requirements. The ability to customize
shared memory allocation allows user specified shared memory segment sizing to meet the needs
of specific environments. This paper describes the extent of the changes concentrating on 32-bit
platforms, and describes how the changes will affect OpenEdge in your particular environment.

OpenEdge Shared Memory Changes

The shared-memory architecture for the OpenEdge RDBMS has changed in two important ways:

First, the method used by the OpenEdge RDBMS to address shared memory is enhanced so
that OpenEdge can now address larger amounts of allocated shared memory on both 32 and
64-bit operating systems.

Second, you can now specify the maximum shared-memory segment size that the OpenEdge
RDBMS will allocate. Previously, the maximum segment size was fixed at 128 megabytes
for 32-bit systems, and 1 gigabyte for 64-bit systems. These enhancements enable the
OpenEdge RDBMS to allocate large shared memory pools using the segment sizes you
specify. However, each operating system imposes additional limitations that the OpenEdge
RDBMS must follow.

OpenEdge RDBMS Release 10 can now address larger amounts of allocated shared memory than
previous releases. On 32-bit systems, the addressable limit is 4 gigabytes and on 64-bit systems,
the addressable limit is 8 terabytes. These limits are architectural limits and are often reduced in
practice by limitations imposed by the underlying operating system and other factors. This is
particularly true on 32-bit systems because the entire process address space is limited to 4
gigabytes and you cannot use all of it for shared memory. In practice, most 32-bit OpenEdge
releases are limited to a maximum of approximately 2 gigabytes of shared memory.

In addition to providing for increased shared memory addressability, OpenEdge Release 10 also
introduces the database configuration parameter, shmsegsize, or Shared Memory Segment Size.
This parameter allows the setting the maximum size of shared memory segments for each
database at database startup time. If shmsegsize is omitted during database startup, OpenEdge
will size shared memory segments automatically based on the shared memory requirements of
the database.
No discussion of shared memory can begin without first considering some operating system
principles, and one of the most important to understand is the concept of the Process Address
Space.

The 32-bit Process Address Space

Many factors affect the total amount of shared memory that the OpenEdge RDBMS can allocate,
including program size, stack space required, shared libraries, memory-mapped files, and
memory allocated from the heap. The address space and memory allocation policies used by
different operating systems are complex and vary considerably from one operating system to
another. Explaining the details of each memory model for each operating system is beyond the
scope of this paper, but the following diagram provides a sample layout for a generic 32-bit
process address space.

Most 32-bit operating systems divide the 4 GB address

space into different address ranges or sections. Each
section has explicit purposes. For example, the
operating system must provide a section of the address
space for the program or the code that is running, and
additional sections for the programs stack and static
data.

Consider the diagram to the left and notice that the

entire 4 GB process address space is divided into
multiple sections including, the kernel, the program
and its data, stack space, heap space used for dynamic
memory allocation, and global data that includes
shared memory and shared libraries or DLLs.

The size of each section varies depending on operating

system and does affect the size of allocated shared
memory segments.

2
The following table summarizes some of the key details of the process address space for Solaris,
Linux, Windows, AIX, and HP-UX. The table shows the total memory allocation per process,
the memory required for the kernel, and the memory available in the user address space.

Operating Total Memory Reserved User Address Space

System per Process Kernel Available
Memory
Solaris 4 GB Kernel and user ~4 GB shared between program, its data & stack, and
space are separated shared memory
into their own 4
GB address spaces
Linux 4 GB 1 GB 3 GB shared between program, its data & stack, and
shared memory

Windows 4 GB 2 GB 2 GB shared between program, its data & stack, and

shared memory

AIX 4 GB divided into One 256 MB One 256 MB segment for the program
sixteen 256 MB segment One 256 MB segment for the programs data & stack
segments Twelve 256 MB segments available for shared memory

HP-UX 4 GB divided into The four quadrants are organized as follows:

four 1 GB 1. 1 GB Program
quadrants 2. 1 GB program data
3. 1 GB global space
4. .75 GB global space

A 32-bit program can allocate and manage a maximum

1.75 GB of shared memory.

Testing the Limits

Progress ran a series of tests to determine the maximum amount of shared memory that
OpenEdge Release 10.1B and Release 10.1C are able to allocate for a single database. Testing
took place on Solaris, Linux, Windows, AIX, HP-UX, and Unixware using the appropriate 32-bit
versions of OpenEdge. A test was considered successful after the database started, a simple ABL
query ran and returned the correct records, PROMON ran and reported correct metrics, and the
database shutdown cleanly.

The table below describes the conditions that define the tests and the resulting maximum-shared
memory size that the 32-bit OpenEdge RDBMS is able to allocate. The test variables include:

The platform including its architecture and operating system release

The OpenEdge release information
The database block size
The startup configuration parameters used when starting the database

The only two database configuration parameters used during testing were B (Database Buffer
Pool) and shmsegsize (Shared Memory Segment Size). Defaults were accepted for all other
parameters. B provides a quick and efficient method to increase the shared memory
3
requirements for the database and the buffer pool is initialized so all of shared memory is
touched. The shmsegsize parameter provides a more efficient way for the operating system
to fit smaller shared memory segments into the address space. By manipulating these two
parameters, the amount of allocated shared memory was adjusted up and down. The maximum
allocated shared memory achieved is shown in the following table.

Operating System OpenEdge Database Shared Memory Allocated

and Hardware Release Information & (as reported by PROMON R&D/Status:
Shared Resources)
Startup Arguments
Solaris 5.10 8K database block size

Sun Sparc - 64-bit 32-bit OE 10.1C B 460000 Total shared memory = 3,884 MB
32 GB Ram for Solaris shmsegsize 512

32-bit OE 10.1B B 480000 Total shared memory = 3,892 MB

for Solaris shmsegsize 512

Linux RH EL 4 U4 8K database block size

Intel x86 - 32-bit 32-bit OE 10.1C B 346000 Total shared memory = 2,807 MB
1,010 MB Ram for Linux shmsegsize 512

32-bit OE 10.1B B 347000 Total shared memory = 2,810 MB

for Linux shmsegsize 512

Windows 2003 Server 4K database block size

Intel x86 - 32-bit 32-bit OE 10.1C B 406000 Total shared memory = 1,711 MB
32 GB Ram for Windows(1,2) shmsegsize 512

Client/server ABL 32-bit OE 10.1B B 420000 Total shared memory = 1,746 MB

connection for Windows shmsegsize 512

Windows 2003 Server 4K database block size

Intel x86 - 32-bit 32-bit OE 10.1C B 350000 Total shared memory 1,477 MB
32 GB Ram for Windows shmsegsize 256

Self-Service ABL 32-bit OE 10.1B B 350000 Total shared memory 1,474 MB

Connection3 for Windows shmsegsize 256

1
OpenEdge for Windows does not support the Windows /3Gb parameter.
2
On 32-bit versions of Windows, the /3GB parameter enables 4 GT RAM Tuning, a feature that enlarges the user-
mode virtual address space to 3 GB and restricts the kernel-mode components to the remaining 1 GB.
( http://msdn2.microsoft.com/en-us/library/ms791558.aspx )
3
The OpenEdge ABL client is a larger program, loads multiple DLLs and has larger data and heap requirements
than the database broker or server. The additional resource requirements reduce the amount of address space
available for shared memory mapping.

4
 Operating System OpenEdge Database Shared Memory Allocated
and Hardware Release Information & (as reported by PROMON R&D/Status:
Shared Resources)
Startup Arguments
AIX 5.2 8K database block size

IBM Power PC - 64-bit & 32-bit OE 10.1C B 340000 Total shared memory = 2,763 MB
IBM Power PC 32-bit for AIX shmsegsize 512
32 GB Ram
32-bit OE 10.1B B 345000 Total shared memory = 2,801 MB
for AIX shmsegsize 512

HP-UX 11.11 8K database block size

HP hp9000 64-bit 32-bit OE 10.1C B 200000 Total shared memory = 1,669 MB

PA_RISC for HP-UX shmsegsize 256
4 GB Ram
32-bit OE 10.1B B 202000 Total shared memory = 1,684 MB
for HP-UX shmsegsize 256

Unixware 7.1.4 8K database block size

Intel x86 Pentium XEON 32-bit OE 10.1C B 350000 Total shared memory = 2,839 MB
2 GB Ram for Unixware shmsegsize 512

32-bit OE 10.1B B 360000 Total shared memory = 2,915 MB

for Unixware shmsegsize 512

Shared Memory Segment Size (shmsegsize) Configuration Parameter

The new database configuration parameter, shmsegsize, or Shared Memory Segment Size, was
introduced to allow tuning of the shared memory segment sizes. The valid values for the
shmsegsize parameter are:

32-bit systems 128, 256, 512, 1024, 2048, 4096 megabytes or

1g, 2g, 4g where g is gigabytes

The 4096 and 4g values are allowed on 32-bits platform, but in reality
a 4 GB, shared memory segment cannot be allocated on 32-bit
platforms. Specifying a shmsegsize of 4096 or 4g will allow the
OpenEdge creation of shared memory segments greater than 2 GB on
systems that support segments greater than 2 GB.

64-bit systems 1024, 2048, 4096, 8192, 16383, 32767 megabytes or

1g, 2g, 4g, 8g, 16g, 32g where g is gigabytes

5
The shmsegsize configuration parameter allows control over the maximum segment size used
by the OpenEdge RDBMS. The actual segment sizes created may be smaller than this due to a
variety of factors such as:

The actual amount of shared memory needed may be less than the requested shmsegsize.
The operating system's maximum segment size may less than the requested shmsegsize.
The amount of available unused address space may be less than the requested shmsegsize.
The value of SHMMAX and other kernel tunables may impose lower limits.

The initial maximum-shared segment size is calculated based on your shared memory
requirements and the support provided by your operating system. If shmsegsize is omitted,
OpenEdge will attempt to fit all the shared memory required into the fewest shared memory
segments possible. Previously the fixed maximum was 128 megabytes for 32-bit systems and 1
gigabyte for 64-bit systems. When supported by the operating system OpenEdge creates fewer
but larger segments. For some operating systems, this enables use of slightly larger database
buffer pools with 32-bit versions of the OpenEdge RDBMS.

The Impact on Your Database

The database data structures placed into shared memory are larger in Release 101B and higher
than they were in prior releases. In particular, shared pointers are now 64 bits wide for both 64-
bit and 32-bit versions of the OpenEdge RDBMS. Previously they were 32 bits wide on 32-bit
versions. As a result, some data structures that are quite numerous, such as database buffer-pool
headers and lock table entries, consume noticeably more memory than before.

While the changes in release 10.1B and later have definite advantages, they are also not without
some drawbacks. Thus, if you have a 32-bit system and you upgrade from an earlier release of
OpenEdge, you might encounter errors when you start the database server or when clients try to
connect to it in self-serving mode because the larger size shared segments cannot be allocated
within the available address space. The following table identifies the potential problems that
could occur and presents their solutions:

Problem Solution

You cannot start the server because data
structures do not fit and they did before.
OpenEdge Message(s) in the database log
file or standard output:
The data space of the process
is not enough for the shm
segment (1176)
Lower the value of B by 10 percent to overcome
this problem if you are close to the maximum
amount of shared memory for the system and the
value of B is large. Continue to reduce the
shared memory requirements for the database
until it starts successfully.

6
 Problem Solution

The server starts, but clients cannot connect Lower the value of shmsegsize. When the
in self-serving mode. This may be caused by address space has several smaller holes, the
lack of sufficient contiguous free address operating system may be able to locate the
space in combination with the larger default segments in them. This is likely to happen in
segment size used by the server. Windows, where the system has mapped .dll's in a
manner such that there are no large chunks of free
Or address space. The problem is similar to filling a
box with variable size blocks of wood. Small
A client that connects to several databases in blocks are easier to fit into the box than larger
self-serving mode may no longer be able to ones.
do so. This can happen due to larger
segments.

OpenEdge Message(s) in the database log

file or standard output:

Unable to attach shared

memory segment name error
##. (1720)

There may be a conflict between large Lower the value of shmsegsize.

shared ABL procedure libraries (that are
mapped in a single contiguous memory
segment) and large shared memory
segments created by the server.

7
Summary

The shared memory addressing enhancements made in OpenEdge Release 10.1B and higher
increases the shared memory addressability on 32-bit and 64-bit platforms and increases the
flexibility of shared memory allocation when the shmsegsize configuration parameter is used.
On 32-bit platforms, the OpenEdge RDBMS will allocate more than 2 GB of shared memory if
the operating systems and machine supports the increased shared memory allocations. Even
though the OpenEdge RDBMS can address larger shared memory pools than it did in prior
releases, it can still only allocate and utilize as much shared memory as the operating system
allows. Understanding the basic principles used by your operating system to build a process
address space will help you understand how to set your configuration parameters when you start
your databases.

References

AIX 5L Version 5.3 General Programming Concepts: Writing and Debugging Programs
SC23-4896-04, Fifth Edition (November 2007)

Solaris Internals, Jim Mauro and Richard McDougall, Sun Microsystems, Inc. 2001

Inside Microsoft Windows Internals, Covering Windows 2000, Windows XP, Windows Server
2003, Mark E. Russinovich, David A. Solomon, Jim Allchin, - Microsoft Press, 2005