This article explains how to set the size of the VxFS Buffer Cache using the correspondent kernel

parameters. Historically, the maximum and minimum values are set to 50% and 5% of the total
physical memory. For current environments, specially when backing storage is provided by SAN
devices that provided their own cache memory, those values may not provided a good balance of
resources.

The article presents some general guidelines on values that can be used to better balance the
available system memory with the buffer cache. Those values are consider starting point and
measurement and judgment must be employed for each system deployment.

Always consult your Independent Software Vendor for recommendations on the optimal
parameters for their application. Use performance measurement tools like Glance or Sar to check
the effect on the system of reducing or increasing the values.

HP-UX 11i v1 (11.11)

In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel
parameters.

# kmtune -q dbc_max_pct -q dbc_min_pct

Parameter Current Dyn Planned Module Version
======================================================================
dbc_max_pct 50 - 50
dbc_min_pct 5 - 5
HP-UX 11i v2 (11.23)

In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel
parameters.
# kctune -q dbc_max_pct -q dbc_min_pct
Tunable Value Expression Changes
dbc_max_pct 50 50 Immed
dbc_min_pct 5 Default Immed

HP-UX 11i v3 (11.31)

According to the "The Unified File Cache (UFC) in HP-UX 11i v3" white paper, HP-UX 11i v3
integrates the Buffer Cache and the Page Cache to provide coherence for file access. As part of the
integration project, kernel interfaces have been implemented to facilitate porting of third-party file
systems.

It has created a new twist to the wide known issue of HP-UX systems with physical memory
footprints larger that 4 GB, "the defaults VxFS file cache are set too high".

This problem can be perceive in two ways:

Glance reports that memory usage is higher that 90%, even in systems with low or no application
usage.
Glance shows that the ksyncer_daemon "trash" the I/O subsystem to 100% for several seconds.
On page 8 of the "The Unified File Cache (UFC) in HP-UX 11i v3" under the Flexible cache flush
ing the ksyncer_daemon operation is explained. Verbatim:

"The ksyncer daemon flushes modified data from the cache to the files at a default interval of 30
seconds. A process can use the fcntl() and fadvise() system calls to override the default and tune
the flush interval at the file level. The ksyncer daemon can be configured (enabled or disabled) by
an entry in /etc/rc.config.d/syncer. However, it is strongly recommended to enable the ksyncer dae
mon (use the default entry) in order to minimize the risk of data loss in the event of a system
failure."

The ksyncer daemon only flushes pages that were modified by the write() access type. A process
can use the msync() system call to flush the pages that were modified by the mmap() access type,
if it needs to ensure that those modifications are written to the file at a certain point in time -
otherwise the modifications are written to the file at a point in time that is unknown to the process,
for example when the pages are reclaimed and freed by the System Pager daemon."

It means that if large amounts of data are in the File Cache memory waiting the be flushed into
non-volatile storage (disk drives or SAN luns), every time the "ksyncer" interval is reach, that data
will be moved to the actual storage, producing an intermittent High write request over the I/O
subsystem.

Reducing the size of the file cache from the defaults 50% maximum / 5% minimum to more
conservative values, will not only reclaim that memory for the system process and reduce the
maximum overall size of data that needs to be flushed from the cache to non-volatile storage, but
will also reduce the risk of data lost due to failure to flush the data in a disaster scenario, like a
power outage.

To determine the current file cache size, use the "kctune" command:
# kctune -q filecache_max -q filecache_min
Tunable Value Expression Changes
filecache_max 1020264448 Default Auto
filecache_min 102023168 Default Auto
Recommended Buffer Cache Limits

The following table can provides a starting point to replace the default values.
Physical Memory Maximum Minimum
<= 4GB 10% 5%
> 4GB <= 8GB 5% 3%
> 8GB <= 16 GB 3% 2%
32+ GB 2% 1%
Is a system reboot required?
A system reboot is not strictly required on HP-UX 11i v2 and 11i v3, since on these releases these
parameters can be dynamically modified. Nonetheless, releasing physical memory already used by
the VxFS Buffer Cache can take several hours depending on the amount of data already cached
and the system memory request patterns, specially if the system is not requesting more memory
for user land applications. Occasionally, due to coalescence problem of the cache memory pages
(memory fragmentation), the Buffer cache memory will not been return to the memory pool in a
timely manner. This may be of special consideration on system with 4 GB of system memory
where total available memory stresses the memory allocation routines.

Is always a good recommendation to schedule down time to apply changes to the Buffer Cache
kernel parameters and reboot the system. That way, newer measures can be taken on a clean slate
that allows to establish a current baseline for further measurements.

Change default values

HP-UX 11i v1
Use the kctune command, "SAM" to set new kernel parameters values. After these changes are
done, reboot the system to immediately reclaim the memory and start a measurement baseline.
System administrators are require to evaluate the cache hit ration (sar -b) do determinate if addit
ional memory can be release or need to be use to improve I/O performance.

HP-UX 11i v2 / v3
Use the kctune command, "SAM" or "System Management Homepage" to set new kernel
parameters values. After these changes are done, reboot the system to immediately reclaim the
memory and start a measurement baseline. System administrators are require to evaluate the cache
hit ration (sar -b) do determinate if additional memory can be release or need to be use to improve
I/O performance.

Glance and VxFS Buffer Cache Usage

Glance information should be taken with a grain of salt when looking for memory utilization.
When displaying memory utilization, Glance take into account the memory assigned to the Buffer
Cache. This can lead to misreadings in some systems with constant I/O usage of VxFS where the
buffer cache holds all the memory provided as their upper limit.

Example:
B3692A Glance C.04.70.001 11:35:04 delta 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SA | 3% 3% 3%
Disk Util | 0% 0% 0%
Mem Util S SU U | 65% 65% 65% (1)(2)
Swap Util U UR R | 31% 31% 31%
--------------------------------------------------------------------------------
PROCESS LIST Users= 10
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 200% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
glance 26717 root 6.0 1 0.0 1.5mb 5.6mb SLEEP
scopeux 1750 root 0.0 1 0.0 120.1mb 125.3mb SLEEP
midaemon 1768 root 0.0 2 0.0 72.6mb 104.8mb

C - cum/interval toggle Page 1 of 1

ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit

1) Observe the Mem Util section. The letters stand for S: System, U; User; B: Buffer. Glance list
the buffer cache as part of the total memory usage. That memory can be released from the
buffer cache by the kernel very quickly when user process request additional memory from
the heap.
2) Is very common to see the "current" and "high" values to reach 90% or above. Always
remove Buffer Cache or File Cache total from that value. To check for that values, press the
"m" key on the Glance TUI and that will lead you to the memory section. More details on
memory usage will be displayed.

B3692A Glance C.04.70.001 11:35:45 delta 9000/800 Current Avg High

--------------------------------------------------------------------------------
CPU Util S | 1% 2% 4%
Disk Util F | 2% 1% 2%
Mem Util S SU U | 65% 65% 65%
Swap Util U UR R | 31% 31% 31%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 10
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 0 111 0.0 2.6 60.6
Page In 0 0 0.0 0.0 5.3
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 0 0 0.0 0.0 0.0
Total VM : 615mb Sys Mem 868mb User Mem: 432mb Phys Mem : 2.0gb (1)
Active VM: 443mb Buf Cache: 4mb Free Mem: 710mb FileCache: 33mb
MemFS Blk Cnt: 0 MemFS Swp Cnt: 0 Page 1 of 1
ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
The last lines of the output display overall system memory utilization with higher detail. Keep in
mind that Buf Cache (File Cache) is part of Sys Mem.

Buffer cache and kmeminfo

The WTEC tool kmeminfo can be very useful to explore the relationships between the different
system memory areas. On system when the default limits and constant I/O will present up to 50%
of memory assigned to the Buffer Cache.
Example:
tool: kmeminfo 9.14 - libp4 9.366 - libhpux 1.258 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.23 64bit IA64 on host "camccscbbur04"
core: /dev/kmem live
link: Fri Nov 09 15:32:57 CST 2007
boot: Mon Oct 19 14:05:05 2009
time: Wed Nov 4 10:43:14 2009
nbpg: 4096 bytes
----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 1041841 4.0g 100% (1)

Free memory = 23333 91.1m 2% (2)
User processes = 161271 630.0m 15% details with -user
System = 849626 3.2g 82%
Kernel = 328706 1.3g 32% kernel text and data
Dynamic Arenas = 117272 458.1m 11% details with -arena
vx_global_kmcac = 28043 109.5m 3%
vx_buffer_kmcac = 17856 69.8m 2%
spinlock = 15145 59.2m 1%
ALLOCB_MBLK_LM = 5440 21.2m 1%
M_DYNAMIC = 5186 20.3m 0%
Other arenas = 45602 178.1m 4% details with -arena
Super page pool = 87939 343.5m 8% details with -kas
Emergency pool = 2075 8.1m 0% system critical reserve
UAREA's = 7392 28.9m 1%
Static Tables = 94475 369.0m 9% details with -static
nbuf = 44896 175.4m 4% bufcache headers
pfdat = 24418 95.4m 2%
vhpt = 8192 32.0m 1%
text = 8084 31.6m 1% vmunix text section
bufhash = 2048 8.0m 0% bufcache hash headers
Other tables = 6836 26.7m 1% details with -static
Buffer cache = 520920 2.0g 50% details with -bufcache (3)

The Physical memory is 4 Gb.

The Free memory is 91 Mb.
The Buffer Cache occupies 2 Gb (50%) of the available system memory, this system will have a
major impact of reducing the size of the Buffer cache. Setting the value to a 10% will release 1.5
Gb of the system memory.

The following article show alternatives to bypass the VxFS file cache for individual VxFS.

One alternative is using OnlineJFS special options options mincache and convosync. The other
alternative is using the vxtunefs command to set a small value to the discovered_direct_iosz param
eter, to bypass I/O request larger than the number.

Most Disk Array | SAN solution provided onboard RAM to accelerate I/O response time, so
bypassing the VxFS buffer cache (11i v1 and v2) or the Unified File Cache (11i v3) may be a
prudent option to save system memory resources otherwise assigned to the file system cache.

Using OnlineJFS parameters

This is done with Online JFS mount options mincache=direct and convosync=direct

Using vxtunefs
By default all I/O greater than 256KB bypasses the cache.
This can be tuned via the vxtunefs parameter discovered_direct_iosz

vxtunefs(1M):
discovered_direct_iosz
Any file I/O requests larger than the discovered_direct_iosz are handled as discovered direct
I/O. A discovered direct I/O is unbuffered like direct I/O, but it does not require a synchronous
commit of the inode when the file is extended or blocks are allocated. For larger I/O requests, the
CPU time for copying the data into the buffer cache and the cost of using memory to buffer
the I/O becomes more expensive than the cost of doing the disk I/O. For these I/O requests, using
discovered direct I/O is more efficient than regular I/O. The default value of this parameter is
256K.