Online Performance Configuration Guidelines for PeopleTools 8.45, 8.46 and 8.47, 8.

48
See Change Record This white paper is a practical guide for technical users, installers, system administrators, and programmers who implement, maintain, or develop applications for your PeopleSoft system. In this white paper, we discuss guidelines on how to diagnose a PeopleSoft Online Transaction environment, including PeopleSoft Internet Architecture and Portal configuration. Configuration of Batch processes is not covered in this document. Much of the information contained in this document originated within the PeopleSoft Global Support Center and is therefore based on "real-life" problems encountered in the field. Although every conceivable problem that one could encounter with Tuxedo, the PeopleSoft Application Server, or your web server is not addressed in this document, the issues that appear in this document are the problems that prove to be the most common or troublesome. This white paper consists provides guidance across the major PIA components: Application Server, Web Server, Web Browser, and OS Kernel. Keep in mind that PeopleSoft updates this document as needed so that it reflects the most current feedback we receive from the field. Therefore, the structure, headings, content, and length of this document are likely to vary with each posted version. To see if the document has been updated since you last downloaded it, compare the date of your version to the date of the version posted on Oracle Metalink or Customer Connection. This paper is not a general introduction to environment tuning and we assume that our readers are experienced IT professionals, with a good understanding of PeopleSoft?s Internet Architecture.To take full advantage of the information covered in this document, we recommend that you have a basic understanding of system administration, basic Internet architecture, relational database concepts/SQL, and how to use PeopleSoft applications. This document is not intended to replace the documentation delivered with the PeopleTools 8 or 8.4x PeopleBooks. We recommend that before you read this document, you read the PIA related information in the PeopleTools PeopleBooks to ensure that you have a wellrounded understanding of our PIA technology. Note: Much of the information in this document eventually gets incorporated into subsequent versions of the PeopleBooks.

Many of the fundamental concepts related to PIA are discussed in the following PeopleSoft PeopleBooks:

PeopleSoft Internet Architecture Administration (PeopleTools|Administration Tools|PeopleSoft Internet Architecture Administration) Application Designer (Development Tools|Application Designer) Application Messaging (Integration Tools|Application Messaging) PeopleCode (Development Tools|PeopleCode Reference) PeopleSoft Installation and Administration PeopleSoft Hardware and Software Requirements

Additionally, we recommend that you read the BEA documentations (in HTML format) delivered with the BEA CD-ROM to gain a thorough understanding of the BEA products that PeopleSoft uses -- Tuxedo, Jolt, and WebLogic Server, as well as documentations from IBM if you plan to deploy WebSphere. Refer to your PeopleSoft Installation and Administration book for directions on accessing the delivered BEA and IBM documentations. This material has not been submitted to any formal PeopleSoft test and is published AS IS. It has not been the subject of rigorous review. Oracle assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a customer responsibility and depends upon the customer's ability to evaluate and integrate them into their operational environments.

TOC/Navigation Title
This white paper contains the following information. 1. Application Server Guidelines 2. Web Server Guidelines 3. Web Browser Configuration 4. Additional Configurations 5. Integration Broker 6. Operating System Settings 7. Call Telephony Interface (CTI) Guidelines 8. PSAE and PSAESRV Guidelines 9. Reporting Tools 10. Monitoring Tools

Application Server Guidelines

Application Server Memory Guidelines

Lack of memory resources on the application is the most common cause of serious online performance issues.If you are experiencing more than 4 seconds of response time for every (as opposed to just a few transactions) web page refresh, it is very likely a problem with application server memory.These problems are most likely caused by "memory swapping". These are the same problems you experience when trying to run Microsoft PowerPoint, Word and Excel on a PC with 12 MB of memory.Here are some things you can do to see if you need more memory for your application server. 1. Make sure don't have too many PSAPPSRV processes booted. During peak usage times, you should see some small amount of queuing for the psappsrv processes. If you have too many processes, this will affect performance. The extra PSAPPSRV processes will not only take up memory space and CPU time from the OS, the main issue is that each PSAPPSRV will not be 'sufficiently' cached. I.e. Each PSAPPSRV can only cache a portion of the user panels. It will take much longer time to fully cache each PSAPPSRV. To determine the queue length of the specific domain, do the following steps: o Run psadmin o Go to ?PeopleSoft Domain Administration? for the specific domain name o Select "TUXEDO command line" o Run the command ?pq? o The ?# Queued? column shows the queue length for that application server process 2. Here is a guideline for the memory footprint for the PSAPPSERV: o CRM and HRMS use about 100-150 MB per app server process. CRM gets 50 users per process, HRMS gets less o Supply chain uses 300-500 MB per process. You get about 10-20 users per process o Financials use 150-300 MB per process. You get about 20-30 users per process So, for example, if you have 1GB of memory available for a financials application server, boot no more than 4 PSAPPSRV processes. 3. Make sure you include all test/dev/prod domains on a computer in your calculation.A common problem is having 4 large domains on one computer with insufficient memory. 4. Check your memory utilization on the appserver when you experience performance problem.Check for swapping by using the memory monitoring procedures outlined in the following sections. 5. If you are swapping, you need to either add more memory or reduce the number of domains or PSAPPSRV process on the computer. Determining PeopleSoft AppServer memory usage under NT Use NT TaskManager to monitor all the PeopleSoft processes. PeopleSoft processes have a process name prefix with ?ps?.

Start TaskManager and select ?View? from the menu bar, and then choose ??Select Columns?. Pick the Memory Usage and Virtual Memory Size columns. Sort the process by name, and the two memory columns will tell you approximately the amount of memory each PeopleSoft process is using. "Memory Usage" is not the entire memory consumption of the process, but only the amount of physical RAM that process is using. However, with NT, an application may be using much more Virtual Memory (i.e., the paging file) than physical memory, so if you're gauging memory at all, you have to have both these fields selected. The memory usage columns will include the shared libraries loaded into each process; therefore, if you add up all the processes, you will be ?double counting? the shared libraries portion. Since PeopleSoft shared a lot of the common libraries, the summation of all PS processes is a quick estimator, not an absolute measurement of memory usage. The total ?Memory Usage? and the ?VM Size? of the combined PeopleSoft processes should not be higher than 70% of the Real memory of the server. Determining if excessive memory swapping happens under NT Start NT Performance Monitor, perfmon. Add the Object Counter, under ?Memory?, and then the ?Page/sec?. Pages/sec is the number of pages read from the disk or written to the disk to resolve memory references to pages that were not in memory at the time of the reference.This is the sum of Pages Input/sec and Pages Output/sec.This counter includes paging traffic on behalf of the system Cache to access file data for applications.This value also includes the pages to/from non-cached mapped memory files. This is the primary counter to observe if you are concerned about excessive memory pressure (that is, thrashing), and the excessive paging that may result. When the Page/sec goes above 100 hard pages per second, there is a swapping problem. The number of PSAPPSERV processes should be reduced or more memory needs to be added to the server. Determining memory usage on UNIX To assist end-user better understand the memory consumption and CPU usage of the PeopleSoft Application Domain, we have created several UNIX vendor specific shell scripts to monitor the above parameters. The monitoring shell scripts can be obtained from Customer Connection web site (https://www.peoplesoft.com/corp/en/login.jsp) Login to the Customer Connection, select Products + Industries on the right. Choose Enterprise Product Lines, select Enterprise Tools and Technology, select Enterprise PeopleTools, select PeopleTools 8.4 underneath. In the middle of the page is the Red Paper

section, which you will find the latest revision of this Red Paper and the Shell Scripts for download. Here is a sample output of the ps_chk_domain script:
Please Enter your application server domain name? PT84 Please Enter interval in seconds? 10 Please Enter # of interval's? 100 Tue Oct 23 23:00:33 PDT 2001 For all PS Processes: Resident memory size is: CPU for PS Processes: Virtual memory size is: 123.543 MB 11 % 198.098MB

For all PSAPPSRV Processes: Resident memory size is: 108.453MB CPU for PS Processes: 9% Virtual memory size is: 106.559 MB

--Resident memory refers to the real physical memory currently required by the process for its operation. Virtual memory refers to the process virtual address size, this include memory that has been paged out to the physical disk. If the Virtual memory continues to increase, the ? RecycleCount? of the AppServer should be lowered. The output of ps_chk_domain.sh divides the portion of PSAPPSRV processes from the entire Domain so that the end-user (or system administrator) can get a better understanding with the distribution of the memory resources. It is recommended that the total Resident memory for the entire ?PS Processes? should not exceed 70% of the total real memory available on the server. Determining if excessive memory swapping happens under UNIX Use vmstat to determine if the OS is swapping. To check whether paging is a problem run vmstat and check out the pi column to see if you are doing much paging in and po for paging out. You can also run sar -q to check paging;or vmstat -s and take multiple samples to see if the page ins and outs to paging space are the majority of the total paging.

Application Server Recycle Count

This configurable parameter dictates the number of services after which AppServer will automatically restart. Background The AppServer processes will use physical runtime memory to cache Panel objects in order to speed up user response time instead of fetching the Panel objects from the database server every time. However, as the number of requests serviced by the AppServer increases, the amount of physical memory occupied by the AppServer processes also increase. When the amount of memory occupied by the AppServer becomes too large (relative to the real memory available at the time), paging to file system will occur and paging will impact user experience. In order to effectively manage the memory footprint of the AppServer, keeping the ? Recycle Count? at a realistic level is important. When the AppServer reaches the specified ?Recycle Count? value, the AppServer will terminate and restart itself. When the AppServer terminated, the occupied memory will be released. Recommendation It is recommended to set Recycle Count at 5000. The ?recycle count? should be adjusted so that no memory swapping is introduced because of a high ?recycle count? value. (See session above for how to determine memory swapping.) To minimize the cost of re-caching the AppServer, it is important to enable ?File Cache? on the AppServer. To enable Server Caching, set the EnableServerCaching=2, (the default value is 2).
----------------------------------------------------------------------; EnableServerCaching ; 0 Server File caching disabled ; 1 Server File caching limited to most used classes ; 2 Server File caching for all types EnableServerCaching=2 -----------------------------------------------------------------------

Application Server Dynamic Recycle (8.48 or above)

Application server restarts are a leading cause of inconsistent response times. If recycle count is set too low, then there will be too many psappsrv restarts. If set too high, there could be instability or crashes as the processes grow too big.

In 8.48, the new Dynamic Recycling feature allows the app server to monitor its own memory growth to recyce when it grows above a certain threshold. The feature is found in psappsrv.cfg in the PSAPPSRV section. You'll see a "Percentage of Memory Growth" option. To enabled this new feature you must uncomment the option, by default the option is commented out (and disabled). Once dynamic recycle is enabled, current 8.48 releases of tools get the virtual size of the psappsrv process every 100 service requests. If any new metadata had been loaded the since the last check then the max memory threshold is reset to be N% larger than current memory usage. Otherwise a check is done to compare the process's current usage to the threshold. When the threshold is exceeded a check is then done to be sure the process has done at least as many requests as the "recycle count" setting. If not enough requests have been processed then message about "delaying recycle" is printed, otherwise the server recycles. This percentage growth option will only extend server service count above the older recycle count setting it does not ever recycle before the old value. For now we recommend customers interested in this feature to try it out in a test environment and see how it behaves and use it to derive a suitable recycle count for the production system.

Shared Cache for Application Server

PeopleTools 8.4x has a shared cache feature in which the various psappsrv processes in an app server domain use a single set of cache files rather than separate cache files for each process. All managed objects are included in the shared cache files so that objects are not loaded from the database. Note: You have to pre-load all the necessary cache objects before you can enable the Shared Cache feature.

PeopleSoft installation supplied with an Application Engine program called LOADCACHE to assist customer in generating this pre-loaded shared cache. Shared Cache provides the following benefits:

Save disk space since all AppServer processes use the same common directory to read the file cache contents

Speed up operation since all the managed objects are cached into the common directory ahead of time, instead of retrieving from the database tables on demand

To run the LOADCACHE program: 1. Make sure that the database that the application server runs against produces clean SYSAUDIT runs. If SYSAUDIT is not clean, the LOADCACHE program may fail. 2. Make sure the user who is executing the program has the ?Load Application Server Cache? component defined to its Permission List in User Profile. Find or add the Utilities page for the users permission list and add the component Load Application Server Cache. 3. (For 8.42 and below only) Check your psprcs.cfg file (Process Scheduler configuration file). psprcs.cfg is where you specify the type of objects to cache using the EnableServerCaching parameter. For PeopleTools 8.4x set it to 2. The LOADCACHE reads this setting and caches metadata according to the value specified in the Process Scheduler configuration. Note. Do not enable shared caching for Process Scheduler. Make sure ServerCacheMode=0
4. 5. 6. 7. 8. 9. [Cache Settings] ;============================================ ; Settings for Tools that use Cache ;============================================ CacheBaseDir=%PS_SERVDIR%\CACHE ;---------------------------------------------------------------------10. ; EnableServerCaching 11. ;0 Server File caching disabled 12. ;1 Server File caching limited to most used classes 13. ;2 Server File caching for all types 14. EnableServerCaching=2 15. ;---------------------------------------------------------------------16. ; CacheBaseDir = the base cache directory 17. ;CacheBaseDir=%PS_SERVDIR%\CACHE 18. ;---------------------------------------------------------------------19. ; ServerCacheMode 20. ;0 One cache directory per App Server Process 21. ;1 Shared Cache 22. ServerCacheMode=0

23. In PeopleTools 8.40 or above: Select PeopleTools, Utilities, Administration, Load Application Server Cache. 24. Enter the appropriate Run Control ID (You may have to add a new value here). The Load Application Server Cache page appears. 25. In the Output Directory specify directory where you want the cached metadata to be written:
Unix example> /ds1/home/testora/pt814c1/PT814U25/appserv NT example> c:\temp\

26. Select the correct process scheduler and click Run. Navigate to? PeopleTools>Process Monitor.Wait for the process to become ?Success?. The first time you run the process may take 4-5 hours. Note: Run the program with Run Location set to ?Server?.

27. Shut down your application server domain. 28. Enable shared caching with the ServerCacheMode parameter (ServerCacheMode=1) in psappsrv.cfg of the domain, and reconfigure the domain so that the changes are reflected.
29. 30. 31. 32. 33. 34. [Cache Settings] ;============================================ ; Settings for Tools that use Cache ;============================================ CacheBaseDir=%PS_SERVDIR%\CACHE ;---------------------------------------------------------------------35. ; EnableServerCaching 36. ;0 Server File caching disabled 37. ;1 Server File caching limited to most used classes 38. ;2 Server File caching for all types 39. EnableServerCaching=2 40. ;---------------------------------------------------------------------41. ; CacheBaseDir = the base cache directory 42. ;CacheBaseDir=%PS_SERVDIR%\CACHE 43. ;---------------------------------------------------------------------44. ; ServerCacheMode 45. ;0 One cache directory per App Server Process 46. ;1 Shared Cache 47. ServerCacheMode=1

48. Create the <PS_HOME>\<DomainName>\cache\share directory for the appropriate domain. 49. Navigate back to your Process Scheduler. You will now have some cache generated under a new 'stage' directory.
Unix example /ds1/home/testora/pt814c1/PT814U25/appserv/CACHE/stage NT example c:\temp\CACHE\stage

50. You should see a bunch of files with *.dat and *.key extensions inside the stage directory. Copy the contents of the stage directory into the share directory on your appserver domain. 51. Reboot your application server domain. Note: For Shared Cache users: Shared Cache is intended to be used in a very disciplined environment. If you chose the use Shared Cache, you should not try to make any application changes that will affect the Meta Data objects, such as Panel definition, Menu structure or even Permission List. It is because once you chose to use Shared Cache, the cache files are all marked as READ ONLY. The AppServer processes cannot updated the cache files for any delta, and each access to the changed Meta Data will result in a database access which is very costly.

The preferred way to use LoadCache files is to copy the entire cache file set to the individual AppServer process (i.e. seeding the cache directory with the pre-built LoadCache fileset). The multiple copies of Cache file in each directory will consume disk space, but it does allow process to update the cache file content. This flexibility will allow you to make application changes and not suffer from performance lost.

Pre-load Cache (8.48 or above)

The LOADCACHE program takes a long time to run and could generate very large files depending on the number of languages enabled and number of components stored in the database. In 8.48, the new pre-load cache feature allows for more fine-grained control on what to cache. There are three steps to configure pre-load cache. All the related menu options are under PeopleTools, Utilities, Administration. 1. Select Pre-Load Components: From this screen, enter a pre-load project name and select the components that you want to pre-load into the file cache 2. Create Pre-Load Project: From this screen, run a batch job to create the pre-load project and wait for the job to finish
3. 4. 5. 6. [Cache Settings] . . . ; Preload Cache projects for file and in memory cache PreloadFileCache=<preload project name>

7. Run Pre-Load Process: Specify the pre-load project name in the PreloadFileCache field in the psappsrv.cfg file. You can then run the preload project from command line using
8. psadmin -c preload -d $DOMAIN

You can also preload cache from a menu option in psadmin.

PSAPPSRV Instances
The number of PSAPPSRV instances should be kept to a low number. PSAPPSRV need to cache PeopleSoft meta-objects in order to be effective. Too many PSAPPSRV process instances will make it difficult to fully cache all of them. This is even more difficult for a Win2000/NT Tuxedo domain because the Bulletin-board process will try to use the same PSAPPSRV process id (not round-robin?ing like in UNIX). The recommendation is to start with 1.5 to 3 PSAPPSRV processes per CPU. If the application server machine is 100% utilized under load, reduce the number of PSAPPSRV processes until there is some idle CPU. If as a result there is significant queueing, then you may need additional application server hardware. In general, it is a good idea to allow queuing to happen during peak working hours. Spawning extra PSAPPSRV instance to handle high-volume workload is not

recommended. Spawning is a relatively new feature in Tuxedo, and is not necessary for running the PeopleSoft application servers. There is actually a high cost incurred when each new PSAPPSRV process is started up. Each process has to establish its cache, which takes time, CPU, and memory. Instead, the "min" setting should be used to indicate how many PSAPPSRV processes are required for proper throughput. To disable spawning, set the Min Instances the same as Max Instances.

Use Parallel Boot

Prior to 8.48, application server processes in a domain need to be started serially. In 8.48, you are given a choice to boot the domain in parallel. We recommend using that option as it significantly reduces the amount of time it requires to bring up a domain. Note: If PeopleSoft Performance Monitor is enabled, parallel boot may not work properly on Linux, AIX, HP/UX and Tru64 and could take as long to boot as serial boot. A potential workaround is to set the following application server JVM option:
-Djava.security.egd=file:/dev/urandom

Set Application Server JVM Options

Customers using functionality that does a lot of processing in Java (e.g. XMLPublisher) should consider setting the JVM options for the application server. The default JVM options may not optimial. In that case, we recommend adding to psappsrv.cfg and/or psprcs.cfg (in addition to the already existing options):
-server -Xms128m -Xmx512m

Web Server Guidelines

Use Jolt Session Pooling (8.48 or above)
In 8.48, Jolt Session Pooling is enabled by default. Without Jolt Session Pooling, each user session requires a dedicated Jolt session between web server and app server, consuming valuable system resources. With Jolt Session Pooling, the Jolt sessions are shared among the users, reducing system resource usage. In our intenal test on a 2-CPU Windows box, we were able to run 500 users with Jolt Session Pooling versus only 280 users without Jolt Session Poolling.

To check Jolt Session Pooling setting, check web.xml under the appropriate directory for each web server platform. To verify if Jolt Session Pooling is enabled, check the web.xml file:
<servlet> <servlet-name>psc</servlet-name> <servlet-class>psft.pt8.psc</servlet-class> . . . <init-param> ... <param-name>joltPooling</param-name> <param-value>true</param-value>

Use Strict Failover and Weighted Load-Balancing (8.48 or above)

Prior to 8.48, when a host fails, the load is redirected to the next host on the server list This could cause system load to increase significantly on that host and make it unstable. With 8.48, you can specify a failover target for each host so you can distribute the failover load. You can also specify a weight for each host so more load will be directed to more powerful machines. To configure strict failover and weighted load-balancing, edit psserver property in configuration.properties. For example
psserver=appserver1:9000#3[appserver99:10010],appserver2:9010#1[appserv er98:10010]

In this case, appserver1 would receive 3x more requests than appserver2. If appserver1 fails, the requests will be routed to appserver99. The load may not always be distributed per the weights, especially when load is light.

Oracle Application Server (Oracle AS)

Set JVM Heap Size to 256MB or higher The Java virtual machine heap space is the memory region where the Java objects (both live and dead) resided. When the Java heap runs out of space, the Java Garbage Collector will be invoked to deallocate unreferenced objects and free up more space for the program to continue its operation. The JVM cannot service user requests during garbage collections.Many customers have their JVM heap size set to a minimum heap size of 64MB and maximum size of 256MB.Setting the JVM heap size to a larger minimum value (preferably equal to the maximum value) avoids the performance hit incurred by dynamically growing the JVM and improves predictability; it also lessens the frequency of JVM garbage collection. To set the heap size for PeopleSoft Internet Architecture open

opmn.xml in <ORACLE_HOME>\opmn\conf for editing. Locate the process-type node for your PIA installation and add
-Xms256m ?Xmx256m ?verbosegc

to the value attribute of the java-options node. The ?verbosegc switch allows you to monitor the amount of heap usage and the time Oracle AS took for garbage collections so you can make further adjustments if necessary. The garbage collection details will be written to the log file <ORACLE_HOME>\opmn\logs\OC4J~<PIA Install Name>~default_island~<jvm instance number>. Depending on which applications you are using, you may need to set the heap size even higher. Capture JOLT Request Timing Traces To collect JOLT request timing traces for an Oracle AS installation of PIA: 1. Using the Custom Properties page of the current WebProfile, add a String property named auditPWD and set the value. This will be the password used in a later step. 2. Stop the webserver. 3. Open opmn.xml in <ORACLE_HOME>\opmn\conf for editing. Locate the process-type node for your PIA installation and add ?-Dloggersize=0? to the value attribute of the java-options node 4. Restart the webserver 5. Before logging on, submit the following URL to reset the existing log: http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>. A screen of messages will appear on the browser window. That's the log that has been collected. 6. Point to the logon URL and logon as usual. 7. After logging on, point to the above URL again (http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>) to retrieve the log. 8. Copy and paste the log from the browser to a text file. (Step 4 - 7 can be repeated for a more comprehensive collection of data) Modify Apache KeepAlive Setting Depending on the expected interval between your users? requests, it may be beneficial to change the Apache KeepAlive Timeout from 15 to 30 seconds and set the Maximum KeepAlive Requests to zero. Open <ORACLE_HOME>\Apache\Apache\conf\httpd.conf for editing. Locate the KeepAlive, MaxKeepAliveRequests, and KeepAliveTimeout lines and set them to ?On?, ?0?, and ?30? respectively.Since KeepAlive controls how long OHS will keep the connection between the client and OHS, it is possible that if there are a large number of concurrent users, increasing KeepAlive from 15 to 30 will negatively impact performance because there will be fewer available connections to service the requests. You should only change it if you see excessive CPU consumption of the OHS process (look for apache.exe in your Task Manager in Windows, or do ps ?ef | grep httpd in UNIX)

and that you expect a stable number of users making requests at interval larger than the default setting of 15 seconds.When you increase KeepAlive, you may need to increase MaxClients or ThreadsPerChild (see the next section) to ensure incoming requests are not starving for connections. Modify ThreadsPerChild and Oc4jCacheSize For Windows, the ThreadsPerChild setting in <ORACLE_HOME>\Apache\Apache\conf\httpd.conf controls the number of concurrent threads (requests) the Oracle HTTP Server can service. In most cases the default value of 150 is sufficient. For UNIX, the equivalent setting is MaxClients. In most cases the default value of 150 is sufficient. For Windows, if you decide to increase ThreadsPerChild, you should set the Oc4jCacheSize to the same value.This setting controls the number of connections between the Oracle HTTP Server and OC4J and setting it to the same value as ThreadsPerChild should give the optimal performance.To change this value, open <ORACLE_HOME>\Apache\Apache\conf\mod_oc4j.conf and search for Oc4jCacheSize.If the entry is not there, add the following line above the first Oc4jMount entry (this assumes you have increased ThreadsPerChild to 150):
Oc4jCacheSize 150

For UNIX, this setting does not need to be changed. Disable HTTP Request Logging in mod_oc4j Open mod_oc4j.conf in <Apache_home>\conf for editing. Locate the appropriate <virtual host> section and comment out the line beginning with TransferLog . Confirm JRE Version JDK 1.4.2 is installed automatically when you install Oracle AS. To check the JVM versions certified with Oracle AS, please login to metalink.oracle.com, click the ?Certify and Availability? menu and click ?View Certifications By Product?. Generate Heap Dumps on OutOfMemoryError If you are using JDK 1.4.2_12 or above, there is a new option to dump the JVM heap if you are running into OutOfMemoryError. You can add ?XX: +HeapDumpOnOutOfMemoryError ?XX:HeapDumpPath=/tmp to the JVM options If there is an OutOfMemoryError, the contents of the heap will be dumped to a file in the directory specified by ?XX:HeapDumpPath.

The heap dump file can be read by the Heap Analysis Tool (HAT), or jhat (in JDK 6), or a profiler that supports the binary heap dump format. It will provide valuable information to Oracle development on what may be causing the OutOfMemoryError.

WebLogic
Confirm Minimum Required Service Pack is installed PeopleSoft Platform Database lists the minimum required service pack for WebLogic Server. %WLS_HOME%/logs/log.txt specifically indicates service pack installed. For PeopleTools 8.46 and 8.47, Service Pack 3 is the minumum version certified and Service Pack 5 is recommend. For 8.48, Service Pack 5 should be used. Under normal circumstances, WebLogic should be using the "Posix Performance Pack", and this will use the native OS's socket implementation. When the "Performance Pack" is not loaded properly, generic socket implementation will be used, which is not efficient and there could be performance issue or socket stability problems. To verify that your WebLogic Server is loading the Performance Pack correctly, check your WebLogic output message and search for the reference to "NT/Posix Performance Pack". Set Thread Count For WebLogic, change the entry in %WLS_HOME%/<peoplesoft web domain>/config.xml as
<Server ListenPort="80" Name="PIA" Notes="The PIA server is the default server for PeopleSoft 8.40 Internet Architecture." TransactionLogFilePrefix="C:\Apps\bea\wlserver6.1/config/peoplesoft/log s/" XMLEntityCache="XMLCacheMBean" XMLRegistry="PeopleSoft IG XML Registry"> <ExecuteQueue Name="default" ThreadCount="N" /> ... </Server>

where N should be based on the ?peak? concurrent HTTP usage at any given point of day, ie, the number of HTTP actions (such as posting a HTTP request or downloading a HTTP response) that can be executed concurrently. This should not include user whom are login but not actively performing any HTTP action. To preserve system resources, administrator can fine tune the number of ExecuteThread to be a percentage of the estimated peak value, such as N = 90% of Peak usage.One way to obtain a more accurate Peak value is to monitor the thread usage via WebLogic console. Note:

For Unix platforms, when you increase the ThreadCount, you are allowing more socket connections to be established. You will need to increase the number of file descriptors (maxfiles and maxfiles_lim) accordingly. To check the current file descriptor value, use the following command: csh ?c ?limit ?h descriptors?

You also need to raise the number of threads limit per process (max_thread_proc) in UNIX. As for Windows these are implicitly limited by other system resources such as memory, and there is no explicit parameter that controls them. Confirm JRE Confirm from BEA's platform page that the installed JRE is certified (may require OS patches). http://e-docs.bea.com/wls/certifications/certifications/index.html Note: For Linux it is necessary to set an environmental variable to work around a JRE bug.
$ export J2SE_PREEMPTCLOSE=1

The rationale behind is the current Linux is still using the old UNIX network/thread semantics, and the close() is not preemptive as in Solaris and AIX. This applies to JRE1.2.2, 1.3 and 1.3.1. Set JVM heap size to 256MB or higher Many customers have their JVM heap size set to a minimum heap size of 64MB and maximum size of 256MB. Depending on which applications you are using, you may need to set the heap size even higher. Setting the JVM heap size to a larger minimum value (preferably equals the maximum value) avoids the performance hit incurred by dynamically growing the JVM and improves predictability; it also lessens the frequency for the JVM's garbage collection. See JVM Heap Size section below to learn how to change the Heap size for specific web server. Set OS File Descriptor to 100*ThreadCount A file descriptor is required for every file that is opened, every *.class file read in by WebLogic, every jolt connection PIA/Portal make to the appserver, every connection that

has to open back to a client, plus any other socket based communication that was occurring on that machine. To raise the file descriptors for UNIX, use the following command:
ulimit ?n 4096

In Windows NT/2000 there is not an explicit parameter for the number of file descriptors. It is implicitly limited by hardware resources, mainly system memory. Lower OS TCP/IP Cleanup/Timeout Settings Socket based applications that are opening and closing hundreds or thousands of sockets need to have sockets they have marked as closed, truly closed. Once a process closes a socket, it is really only marked as closed until the OS, based on a cleanup/flush timeout, makes that socket available again. The default value for this is 11 minutes, lower this value to 1 minute. Planet PeopleSoft implements 1 minute TCP wait time. Information on this is provided by BEA as well. See the Reducing TCP Wait Time section to learn how to change the TCP Wait Time for a specific OS. Monitor JVM Garbage Collection The Java virtual machine heap space is the memory region where the Java objects (both live and dead) resided. When the Java heap runs out of space, the Java Garbage Collector will be invoked to deallocate the dead objects and free up more space for the program to continue its operation. The JVM cannot service user requests during garbage collections. To monitor the amount of heap usage and the time WebLogic took for the Garbage Collection, you can add the verbosegc switch to the setEnv.cmd script file. You have to start the WebLogic from the command line -- startPIA.cmd (instead of an NT service) to see the GC output. In setEnv.cmd,
SET JAVA_OPTIONS=-hotspot ?Xms256m ?Xmx256m ?verbosegc

Here is a sample output of the GC:

Sat Nov 24 22:15:34 PST 2001:<I> <WebLogicServer> Invoking garbage collection Sat Nov 24 22:15:34 PST 2001:<I> <GC> GC: Before free/total=46867368/67108856 (69%) <GC: freed 249213 objects, 15440712 bytes in 396 ms, 95% free (51334096/53687088)>

<GC: init&scan: 6 ms, scan handles: 105 ms, sweep: 124 ms, compact: 161 ms> <GC: 0 register-marked objects, 140 stack-marked objects> <GC: 1 register-marked handles, 559 stack-marked handles> <GC: refs: soft 0 (age >= 32), weak 0, final 559, phantom 0> <GC: compactHeap: blocks_moved=249506> <GC: 0 explicitly pinned objects, 35 conservatively pinned objects> <GC: last free block at 0x02A11B2C of length 35906768, is at end>

To keep the performance degradation from Garbage Collection at a minimum, users should use the command line option -noclassgc. This will inhibit a thread that would normally clear out unused classes (thus saving the load incurred by that thread). The goals of tuning your heap size are twofold: minimize the amount of time that you spend doing GC while maximizing the amount of clients that you can handle at a given time. Disable Servlet Reload In %WLS_HOME%/<peoplesoft web domain>/config.xml there is a parameter ? ServletReloadCheckSecs ? that dictates how often WebLogic checks whether a servlet has been modified, and if so reloads it: <WebAppComponent Name="PORTAL" ServletReloadCheckSecs="-1" Targets="PIA" URI="PORTAL" /> In a production environment the servlets are not modified so checking and reloading would only incur unnecessary work, and thus ServletReloadCheckSecs should be set to ??1? for each of the components. (If the field is not present it defaults to 0 ? always reload, which is undesirable. In that case the field should be introduced with value ?1.) Capture JOLT Request Timing Traces 1. Using the Custom Properties page of the current WebProfile, add a String property named auditPWD and set the value. This will be the password used in a later step. 2. Stop the webserver. 3. In the weblogic domain directory, open startPIA.cmd/startPIA.sh and add a command-line option -Dloggersize=0 to the firing of the java process, like the following:
%JAVA_HOME%\bin\java" %JAVA_OPTIONS% -classpath %CLASSPATH% -Dweblogic.Domain=%DOMAIN_NAME% -Dweblogic.Name=%SERVER_NAME% -Dbea.home="%BEA_HOME%" -Dweblogic.management.password= %SYSTEMPASSWORD% -Dweblogic.ProductionModeEnabled=%STARTMODE% "Djava.security.policy==%BEA_HOME %/wlserver6.1/lib/weblogic.policy" -Dweblogic.management.discover=%DISCOVERY_MODE% -Dloggersize=0 weblogic.Server

4. Restart the webserver

5. Before logging on, submit the following URL to reset the existing log: http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>. A screen of messages will appear on the browser window. That's the log that has been collected. 6. Point to the logon URL and logon as usual. 7. After logging on, point to the above URL again (http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>) to retrieve the log. 8. Copy and paste the log from the browser to a text file. (Step 4 - 7 can be repeated for a more comprehensive collection of data) Reduce Unnecessary Logging Set logging to false by default in config.xml:
WebServer DefaultWebApp="PORTAL" LogFileBufferKBytes="64" LogFileName="C:\Apps\bea\wlserver6.1/config/peoplesoft/logs/PIA_access. log" LoggingEnabled="false" MaxLogFileSizeKBytes="50000" Name="PIA" WAPEnabled="true" />

WebSphere
Setup and Configuration Note: PeopleTools 8.4 supports native Http Server within WebSphere at port 9080 and 9443, for http and https, respectively. Use of IBM Http Server, which listens at port 80 or 443 for http or https, is optional Note: Before you begin the installation, review the PeopleSoft Platforms Database to make sure you are installing WebSphere into a supported environment.

For detailed installation instructions, especially the required e-fixes needed for WebSphere please refer to the ?WebSphere Install? document in Customer Connection. Note: Install WebSphere eFixes.The user can install eFixes in any order. To remove eFixes rollback efixes in the reverse order (that you applied).

Set Thread Count In %WAS_HOME%/config/server-cfg.xml, locate the following line (In Websphere 5 , The configuration file is located at %WAS_HOME %\config\cells\\nodes\\servers\server1\server.xml):
<webContainer xmi:id="WebContainer_1" installedWebModules="WebModuleRef_1 WebModuleRef_2 WebModuleRef_3 WebModuleRef_9 WebModuleRef_10 WebModuleRef_11"><threadPool xmi:id="ThreadPool_1" minimumSize="n" maximumSize="N" inactivityTimeout="100" isGrowable="false"/></webContainer>

where N should be based on the ?peak? concurrent HTTP usage at any given point of day. The number of HTTP actions (such as posting a HTTP request or downloading a HTTP response) that can be executed concurrently. This should not include user whom are login but not actively performing any HTTP action. For WebSphere 5, the tags have changed so please consult the WebSphere documentation on the correct attribute to modify. To preserve system resources, administrator can fine tune the number of Thread to be a percentage of the estimated peak value, such as N = 90% of Peak usage Confirm JRE Note: When you install WebSphere, IBM?s JRE 1.3 is installed automatically. For WebSphere 5, JRE 1.4.1 is installed

Set JVM Heap Size to 256MB or higher and Monitor JVM Garbage Collection (Refer to the WebLogic section on how to decide the JVM heap size.) In %WAS_HOME%/config/server-cfg.xml, locate the following lines:
<jvmSettings xmi:id="JavaVirtualMachine_1" classpath="$ {WAS_ROOT}/lib/bootstrap.jar;${WAS_ROOT}/properties;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entappletbase.jar;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entapplethttp.jar;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entappletp10.jar;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entappletp12.jar;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEB-INF/lib/entappletp5.jar; ${WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entappletp7.jar;$ {WAS_ROOT}/installedApps/peoplesoft/PORTAL/WEBINF/lib/entappletssl.jar" bootClasspath="" verboseModeClass="false"

verboseModeGarbageCollection="false" verboseModeJNI="false" initialHeapSize="256" maximumHeapSize="256" runHProf="false" hprofArguments="" debugMode="false" debugArgs="" genericCommandLineArgs="com.ibm.ws.runtime.StandardServer" disableJIT="false"></jvmSettings>

Note that verbosegc can also be turned on here (verboseModeGarbageCollection). With IBM websphere 5, you can go to administration console (http://:9090/admin/) to make setting changes. Login with blank user id. On the left Navigation , click on Servers -> Application Servers. Click on your server name (default is server1). Click on Process Definition under Additional Properties. Click on Java Virtual Machine under additional Properties. Here you can change and all the parameters you need verbosegc etc. Other JVM Options Other JVM options, if needed (e.g. -noglassgc), should be inserted in the server-cfg.xml file rather than the startup script (startserver). For example,
<processDefinition xmi:type="server:JavaProcessDef" xmi:id="ProcessDef_1" executableName="${JAVA_HOME}/bin/java" commandLineArguments="-noclassgc" workingDirectory="${WAS_ROOT}/bin" executableTargetKind="JAVA_CLASS" executableTarget="com.ibm.ws.bootstrap.WSLauncher"></processDefinition>

Disable Servlet Reload The servlet reload parameters in WebSphere is located in %WAS_HOME%/<peoplesoft web domain>/[PORTAL/PSIGW/PSINTERLINKS]/WEB-INF/ibm-web-ext-xmi (usually this is the first line): (In websphere 5, the xmi file is located at %WAS_HOME %\AppServer\config\cells\\applications\peoplesoftWAS.ear\deployments\peoplesoftWAS\ PSINTERLINKS\WEB-INF)
<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmi:id="WebAppExtension_1" reloadInterval="0" reloadingEnabled="false" fileServingEnabled="true" directoryBrowsingEnabled="true" serveServletsByClassnameEnabled="false">

(In the PSINTERLINKS/WEB-INF directory such file may not exist. In that case create the file ?ibm-web-ext.xmi? with the following content:
<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmi:id="WebAppExtension_1" reloadInterval="0" reloadingEnabled="false" fileServingEnabled="true" directoryBrowsingEnabled="true" serveServletsByClassnameEnabled="false"> <defaultErrorPage xsi:nil="true"/> <additionalClassPath xsi:nil="true"/> <webApp href="WEB-INF/web.xml#WebApp_1"/> <extendedServlets xmi:id="ServletExtension_1"> <extendedServlet href="WEB-INF/web.xml#Servlet_1"/> </extendedServlets> </webappext:WebAppExtension>

Also copy the file ?ibm-web-bnd.xmi? from the PORTAL/WEB-INF directory.) Moreover, %WAS_HOME%/<peoplesoft web domain>/META-INF/ibm-application-extxmi should look like this (with the reloadInterval line deleted):
<applicationext:ApplicationExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:applicationext="applicationext.xmi" xmlns:application="application.xmi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmi:id="Application_ID_Ext" reloadInterval="0"> <!-- DELETE THIS LINE <reloadInterval xsi:nil="true"/>DELETE THIS LINE --> <application href="META-INF/application.xml#Application_ID"/> </applicationext:ApplicationExtension>

Refer to the same topic in WebLogic for further information. Capture JOLT Request Timing Trace Here is the instructions on how to collect JOLT request timing traces for WebSphere: 1. Using the Custom Properties page of the current WebProfile, add a String property named auditPWD and set the value. This will be the password used in a later step. 2. Stop the webserver. 3. In %WAS_HOME%/config/server-cfg.xml, locate the following lines:
<processDefinition xmi:type="server:JavaProcessDef" xmi:id="ProcessDef_1" executableName="${JAVA_HOME}/bin/java" commandLineArguments="-Dloggersize=0" workingDirectory="$ {WAS_ROOT}/bin" executableTargetKind="JAVA_CLASS" executableTarget="com.ibm.ws.bootstrap.WSLauncher">

4. Restart the webserver 5. Before logging on, submit the following URL to reset the existing log: http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>. A screen of messages will appear on the browser window. That's the log that has been collected. 6. Point to the logon URL and logon as usual.

7. After logging on, point to the above URL again (http://<hostname:port number>/psp/ps/?cmd=resetlog&pwd=<password from step 1>) to retrieve the log. 8. Copy and paste the log from the browser to a text file. (Step 4 - 7 can be repeated for a more comprehensive collection of data) Using Resource Analyzer The Resource Analyzer is a new tool for the Advanced Single Server Edition of WebSphere Application Server. For information on using the Resource Analyzer, refer to the Version 4.0 WebSphere Application Server InfoCenter for the Advanced Edition at http://www.ibm.com/software/webservers/appserv/infocenter.html. To use the Resource Analyzer with the Advanced Single Server Edition of WebSphere, do the following: 1. Install the PmiSingleServerBean.ear file (available under the %WAS_HOME %\installableApps directory) 2. Go to the %WAS_HOME%\bin directory. 3. Enter the command:
seappinstall-install ..\installableApps\PmiSingleServerBean.ear -ejbdeploy false

4. Press Enter when prompted for the Default data source JNDI name and the JNDI name. This allows the use of default values as required by the Resource Analyzer. 5. Restart the application server. 6. Start the Resource Analyzer: 7. Go to the %WAS_HOME%\bin directory. 8. Enter the command: ra hostname 900 AES

Web Browser Configuration

Take advantage of the browser?s caching ability to reduce unnecessary checking and downloading of the same Internet objects within the online transaction session. The Internet objects that most likely remain unchanged during an online session are:

Graphics objects Style Sheets Java Scripts Some HTML pages (such as PIA Login page)

The following web browser settings are the recommended configurations. These configurations are usually the default settings of the respective browsers. Please verify to ensure correctness.

Microsoft Internet Explorer

Here is the configuration recipe: ?Internet Option?->?Temporary Internet Files?-> ?Settings?, ?Automatically?.

Netscape Browser
Here is the configuration recipe: ?Edit?->?Preferences?->?Advanced?->?Cache?->?Once per session?

HTTP 1.1 Compliant Web Browser

Only HTTP 1.1 compliant web browsers request compressed files. Web browsers that are not HTTP 1.1 compliant request and receive the files un-compressed, therefore the Web Server will not be able to send any compressed HTML file to the browser (despite the configuration.properties file has the Compress Response switch turns on.) Most newer browsers since 1998/1999 have been equipped to support the HTTP 1.1 standard known as "content-encoding." Essentially the browser indicates to the server that it can accept "content encoding" and if the server is capable it will then compress the data and transmit it. The browser decompresses it and then renders the page. This is an important performance feature for the Web Browsers that are using the slower bandwidth medium, such as dialup connections or 56K lines. Which Browsers are HTTP 1.1 compliant?

Netscape 4.5 and above, including Netscape 6 and 7. Internet Explorer versions 4 and above (other than IE Mac versions 4.5/5 which do not support HTTP 1.1 compression).

Here is the configuration recipe to verify if Internet Explorer is configured to use the HTTP 1.1 protocol: 1. 2. 3. 4. 5. Open the Internet Options property sheet If using IE 4, this is located under the View menu If using IE 5 or above, this is located under the Tools menu Select the Advanced tab Under HTTP 1.1 settings, verify that Use HTTP 1.1 is selected

Additional Configurations
Browser Compression

From 8.44 on, you can enable compression between web server and browser by selecting the "Compress Response" checkbox via PIA. Go to PeopleTools->Web Profile->Web Profile Configuration, select the Web Profile you are using and look for the "Compress Response" checkbox. Unselecting it turns off compression. Gzip and Compress are supported. The checkbox is selected by default.. If the "Compress Response References" checkbox is selected, compression of additional resource files from web server to the browser is enabled. Only files that have their mimetypes specified in the "Compress Mime Types" field will be compressed. Gzip and Compress are supported. The checkbox is deselected by default because there were some versions of Internet Explorer with certain settings that didn't like javascript and css in compressed format. However, that is a rare scenario therefore in general it should be set to selected. The "Compress Mime Types" text field specifies a comma-delimited list of the MIME-type objects that should be sent in compressed form to the browser. Note that this is applicable only when "Compress Response References" is set to true. Gzip and Compress are supported. Default: application/x-javascript,text/javascript,text/css,text/html. Note: The main purpose of compression is to reduce the amount of data to be transmitted, but that comes with a price ? the extra CPU processing time. In cases where high-speed links are used and the gain in transmission time does not justify the loss in CPU processing time, compression should be turned off. Note: For PeopleTools up to 8.41 the compression settings would cause an error for certain update of Internet Explorer 6.0. In particular, the PIA menu may disappear when a user opens a PIA page. So it is advised to turn them off. This is fixed in PeopleTools 8.42 onwards. Note: Some query entries are truncated when IE tries to open queries in Mircosoft Excel. A workaround (for 8.43 and above) is to turn off query compression by deselecting the "Compress Query" checkbox. This parameter is to turn off JUST queries.

Caching

Caching improves system performance by reducing service requests from the web server to the application server. From 8.44 on, cache settings are configured via PIA. Go to PeopleTools->Web Profile->Web Profile Configuration, select the Web Profile you are using and click the "Caching" tab. If "Cache Portal Objects" is checked, the portal servlet (psp) will cache the following objects in web server memory:

Portal Registry Node (Remote and Local) Content Reference Template (Static only)

If "Cache Portal Objects" is checked, object changes won't take effect until the objects become stale, and are refreshed (see Cache Stale Interval setting below), or the web server is restarted. The checkbox is selected by default "Cache Stale Interval" is the amount of time, in seconds, before portal cache is considered stale, and updated with the latest copy from the application server.In other words, this is the amount of time before changes to cached objects take effect. This setting applies to the same objects as "Cache Portal Objects". Default setting: 86200 (24 hours) The portal automatically throws away all cache entries in memory after the number of requests specified in "Cache Purge All Hit Count". This setting applies for all web sites on this web server. Setting this value to -1 disables hitcount purging. Default setting is 1000 The portal will cache proxied javascripts to improve performance if the "Cache Proxied JavaScripts" checkbox is selected. The checkbox is selected by default. Target content is cached in memory when the TargetContent tag in the template specifies that the target should be cached. Only static content should be displayed in a template with a cached target tag. The TargetContent tag should look like this:
<TargetContent Name="TransactionContent"><Cache Scope="application" Interval="1200" >dummy</Cache></TargetContent>

Pagelets can be cached using the same mechanism. The "Cache Target Content" checkbox should be selected to allow caching of target content. De-selecting it disables all target content caching in the portal servlet, even if the target tag specifies cached content. This checkbox is selected by default. Homepages can also be cached on each user's browser. This means that the browser does not access the web server after the homepage is initially retrieved. You can turn this feature on or off, and also adjust the specific time interval that must pass before the web server is accessed again to get a "fresh" homepage. In any case, if a user clicks the browser's "refresh" button, the homepage is accessed from the web server again, overwriting the homepage cached on the browser.

Caching the Homepage should be beneficial for either production or development environment. It is recommended to have Homepage cache turned on. The following table lists default values of the parameters in PIA: Cache-related properties and default values Cache Homepage=selected Homepage Stale Interval=1200 "Homepage Stale Interval" is measured in seconds. Note: The "Browsers" section specifies which web browser could be used with homepage caching. Please verify to make sure your choice of browser is enabled for caching.

Navigation Pages Caching

As with homepages, navigation pages can be cached on each user's browser. This will improve user response time in traversing between cached menu pages. The Portal Administrator can set systemwide options for navigation cache by using the "Define Personalizations" page and modifying the value for METAXP. It is recommended to set the METAXP to a high value. This will keep the menu pages inside the browser cache. If your menu pages are not going to change frequently, set METAXP to 10080 (one week) or more. Go to PeopleTools-> Personalization->.Personalization Options. Type ?PPTL? (PeopleTools) as the Option Category level value. Click on the Format tab and then click on Set Option Default Value in the METAXP row. Key in the appropriate value and click OK. Then click save when being brought back to the previous screen. Note that a change to METAXP is picked up by the application server immediately, however, since the users? browsers already have cache control set by the previous value of METAXP, browser cache has to be deleted for the new METAXP to take effect. A user can override the value of METAXP for his/her own browsing sessions: after logging on, go to My Personalizations, then hit the Personalize Option button for General Options. Change the METAXP value by entering a new value for Time page held in cache and then click OK. Navigation Caching should not be used during development time because any newly added menus will not be reflected in the cache. Navigation cache should only be used in production systems.

HTTP KeepAlive
HTTP KeepAlive is intended to maintain a persistent socket connection between the Web Browser and the Web Server so that no new connections are required when the HTML pages refer to other HTML objects. However, keeping the socket connection persistent will occupy a socket pair between the Browser and the Web Server. When the keepalive timing is set too long, the following problems are created: The Web Server will need to manage many idle connections Keeping too many socket handles around will reduce the number of sockets available for new connection use. It is generally advised that in 8.4x, keepalive should be turned on, which is the default for WebLogic, WebSphere, and Oracle Application Server.

Reducing TCP Wait Time

The default TCP Wait Time for both NT and most UNIX operating systems are 4 minutes, which is too long for PIA usage and tends to leave too many socket connections staying in the TIME_WAIT state. By shortening the TCP Wait time, the socket can be recycled more efficiently For Windows NT Use the registry editor, regedit, and create a REG_DWORD named TcpTimedWaitDelay under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TcpIp\Parameters Set the value to 60 secs (this one is measured in secs). For AIX To see the current TCP_TIMEWAIT value, run the following command:
/usr/sbin/no ?a | grep tcp_timewait

To set the TCP_TIMEWAIT values to 15 seconds, run the following command:

/usr/sbin/no ?o tcp_timewait =1

The tcp_timewait option is used to configure how long connections are kept in the timewait state. It is given in 15-second intervals, and the default is 1. Using the default or set it to 2 for slower network is recommended. Note: Be careful when you use this command. The no command performs no range checks, therefore it accepts all values for the variables. If used incorrectly, the no command can cause your system to become inoperable.

This command only operates on the currently running kernel. The command must be run again after each startup or after the network has been configured. For HP/UX 11 or Solaris 2.8 and above As root do:
ndd -set /dev/tcp tcp_time_wait_interval 60000

where 60000 is for 60 secs, the minimum recommended value. You must put this command in one of the rc2.d scripts to get it to be automatically set at boot time. For HP/UX 11, you can place the ndd settings inside the file /etc/rc.config.d/nddconf without the need of the startup scripts. For Solaris prior to 2.8 Use the parameter name tcp_close_wait_interval instead of tcp_time_wait_interval.

AIX Thread Model

IBM has recommended the following environment variables setup to improve the Threading model with PeopleSoft?s PIA. Set up the following environment variables for the PeopleSoft Application Server user, database user and the Web Server user: For Korn Shell user, please place the following lines in the .profile:
export export export export export AIXTHREAD_SCOPE=S AIXTHREAD_MNRATIO=1:1 AIXTHREAD_COND_DEBUG=OFF AIXTHREAD_GUARDPAGES=4 AIXTHREAD_MUTEX_DEBUG=OFF

export AIXTHREAD_RWLOCK_DEBUG=OFF

For C Shell user, please place the following lines in the .cshrc file:
set set set set set set AIXTHREAD_SCOPE = S AIXTHREAD_MNRATIO = 1:1 AIXTHREAD_COND_DEBUG = OFF AIXTHREAD_GUARDPAGES = 4 AIXTHREAD_MUTEX_DEBUG = OFF AIXTHREAD_RWLOCK_DEBUG = OFF

Timeout Settings
There are in general three types of timeouts: 1. Timeout during a PIA transaction (Intra-Transactional) ? When the timeout expires, the transaction will fail and need to be resubmitted. 2. Timeout between PIA transaction (Inter-Transactional) ? When the timeout expires, the user has been idling for too long, and the resources associated with the user will be freed. Need to relogin to re-establish the state. 3. Timeout that "reserves" a resource until the expiration time before putting it back into the pool for recycling. (e.g. tcp_timewait) It should be noted that for the Intra-transactional timeouts, their values should be ?staged?. In other words, the end-to-end timeout value should always be greater than that of its intermediate leg(s). With this in mind we can look at our timeout values, from bottom up:

Appserver: Service Timeout = x sec (default 300) (in psappsrv.cfg) Type=Intra-transaction, and it includes the time it takes at the DB server. According to the PIA Answer Book:

Service Timeout Specifies the number of seconds a PSAPPSRV waits for a service request, such as MgrGetObj or PprLoad to complete, before timing out. Service Timeouts are recorded in the TUXLOG and APPSRV.LOG. In the event of a timeout, PSSAPSRV terminates itself and Tuxedo automatically restarts this process. In other words it has to be large enough to accommodate the longest acceptable requests and queries It is recommended to set x=1200 (20 minutes).

PIA: tuxedo_send_timeout = w sec (default 50) ; tuexdo_receive_timeout = x sec (default 600) (in pstools.properties)

The receive timeout is also intra-transactional, and it has to be bigger than appserver service timeout. The receive timeout indicates the maximum number of seconds that the servlet waits for a response from the application server. If you increase your application server service timeouts, such as the Service Timeout setting for PSAPPSRV, then increase the tuxedo_receive_timeout parameter to be greater than the Service Timeout values that appear in the PSAPPSRV.CFG configuration file on the application server.

Servlet: sessionTimeout = x sec (default 1200) (in configuration.properties) This is inter-transactional, as one has to re-login when the servlet expires (technically this is a security setting). The meta refresh tag in seconds. It should be less than or equal to the session.timeout for the servlet. For WebLogic 6.1 it should be less than or equal to <session-timeout> as discussed below.

JOLT: Client Cleanup Timeout = x min (default 60) (in psappsrv.cfg) This is inter-transactional. See description (default is 60, but in most cases this can be reduced to 10 to conserve resources): Client Cleanup Timeout: Specifies the amount of time, in minutes, that a client connection can remain idle (no work requested) before Tuxedo will terminate a client connection. Client disconnects are transparent to a client, and a user just needs to click the mouse to cause a reconnection.

Webserver session timeout:

<session-config><session-timeout>x</session-timeout></sessionconfig>

(in <webserver home dir>/<peoplesoft web domain>/PORTAL/WEBINF/web.xml) This is inter-transactional and specifies the number of minutes (WebLogic) to wait before invalidating an unused session. Note that setting this value too high would tie up webserver resources, especially when users close the browser instead of logging out gracefully. Setting it to be the same as (or a bit higher than) the JOLT cleanup timeout is generally a good idea.

HTTP timeout:

Apache: Timeout = x sec (in httpd.conf) This, unfortunately, serves as both inter-transactional and intra-transactional in different scenarios, so it may or may not be higher than the rest of the timeouts.This directive defines the amount of time Apache will wait on three occasions: 1. the total amount of time it takes to receive an HTTP GET request; 2. the amount of time between receipt of TCP packets on a POST or PUT request; 3. the amount of time between acknowledgements on transmissions of TCP packets in responses. A common problem: PIA: Time out Errors on PeopleSoft pages on PIA even when people are using the page. Solution: 1. Increase the timeout values in servlet session timeout AND webserver session timeout. In configuration.properties:
sessionTimeout = 3600 sec

In web.xml:
<session-config> <session-timeout>60</session-timeout> </session-config>

2. Increase values in configuration.properties in Apache Group>Apache\htdocs\PeopleSoft "meta-tag" session timeout to be 3600 (This will allow users to use the page for 60 minutes with no time out errors) 3. Increase the values in pstools.properties if long queries are common: tuxedo_receive_timeout to 1500

Integration Broker
Configure Load Balance Interval (8.48 or above)

During on-idle processing, the Integration Broker dispatcher will ping nodes that are down (Publication Dispatcher only) and also synchronizes its in-memory queue with the database queue table. If the dispatcher is constantly busy, the on-idle processing may not get run for extended periods of time, resulting in failed messages not getting retried or new messages not getting processed. In 8.48, there is now a new configuration parameter in psappsrv.cfg called "Load Balance Interval". Setting this parameter (measured in minutes) will force the dispatcher to perform on-idle processing at fixed interval, which could help avoid queues not getting processed due to the above reasons.

Master/Slave
When setting up a master/slave configuration using machines with different CPU processing power, use the more powerful machine as the master. If the slower machine is used as the master, it may not keep up with the slave and hence the slaves would be sitting idle, resulting in throughput not being maximized.

Reducing Maximum App Message Size

Some operations may benefit from smaller message size. For example, when running FULLSYNC message from a HCM instance to setup Enterprise Portal Resource Finder, reducing the maximum app message size from default of 10MB to 25K makes the whole process runs significantly faster. To reduce the maximum app message size, go to PeopleTools->Utilities->Administration Options-> PeopleTools Options and change the "Maximum App Message Size" field. We suggest setting the value back to the default after the FULLSYNC messages are completed as this setting will affect all message types.

MultiThreaded Integration Broker (PeopleTools 8.46 or higher)

Synchronous Messaging: PeopleTools 8.46 provides a multithreaded version of IntBroker.SyncRequest() and allows the user to configure the number of message handling threads. In general, setting the thread count equal to the number of psappsrv.exe processes in the target domain for the messages can improve message handling response times by as much as 80% over a single threaded implementation. Note: Make sure you apply 8.46.12 or 8.47.06 if you are not on 8.48. These patches fixes an important thread leak issue associated with using this feature.

Asynchronous Messaging: As with synchronous messaging it?s recommended to set the number of threads for PSPUBHND equal to the number of psappsrv processes in the target

domain. Note: If needed PSPUBHND can now be distributed via a Master/Slave configuration if the number of threads is limited by the source domain hardware.

Scan Time Setting for App Messaging

For Oracle database user who has database logging turn on may see large volume of archive log created. The contents of the Oracle Archive log contained the following:
SET TRANSACTION READ WRITE; INTERNAL; COMMIT;

The Archive Log (or Redo log) was a side effect created by the Application Messaging Dispatchers? select statement, which is invoked periodically based on the value of the ? Scan Interval?. Application Messaging has been enhanced to minimize the side effect on Oracle logging.

PUBSUB Error And App Server Log File Growing

Problem: These PSSUBDSP_dflt errors were occurring upon starting the App server. There are hundreds of entries like this. While these errors occurred, no access to the PS system was possible in 3-tier mode. 2-tier mode was fine.
PSSUBDSP_dflt.1784 [10/24/00 13:46:22](1) (NET.333): The service PSSUBHND_dflt could not be sent because of a blocking condition within Tuxedo on the client. PSSUBDSP_dflt.1784 [10/24/00 13:46:22](1) (NET.333): The service PSSUBHND_dflt could not be sent because of a blocking condition within Tuxedo on the client. PSSUBDSP_dflt.1784 [10/24/00 13:46:22](1) (NET.333): The service PSSUBHND_dflt could not be sent because of a blocking condition within Tuxedo on the client.

Second Error
>PSSUBHND_dflt.2253 [10/24/00 13:46:25 SubConProcess](1) (NET.346): Failed to execute SubConProcess request PSSUBHND_dflt.2253 [10/24/00 13:46:25](2) Service SubConProcess failed

Resolution: This problem only appears when the database is shutdown while the appserver is running.The nightly backup was inadvertently doing this.We tested shutting down

everything in the correct order and this problem does not occur.The PSSUBHND instances are too busy erring out to handle even one subscription. The only way to fix this is to shut everything down (except for the OS) and restart it in the correct order.For some reason, you can shut down the appserver and process scheduler (not the DB) and re-start them.This does not fix the problem, though this might be a combination of the fact there is an under-configured PSSUBHND.

Operating System Settings

For a large environment with many domains the following kernel parameters will provide a good deal of resources. Kernel parameters this size is not uncommon in our customer environments.

For Linux
The default settings and available tuning options vary from one Linux kernel version to the next, so information presented in this section may be incomplete. The correct settings will also vary depending on things like available physical memory, number of processors, and PeopleTools component configuration. <H4< for settings> These settings can be found in the file /etc/sysctl.conf which is loaded when the system boots.
# raise max open fd limits fs.file-max = 65536 # use wider range for local ports, # setup all Tools components to use port numbers between 1024 and 10000 net.ipv4.ip_local_port_range = 10000 65000 # allow more/larger IPC message queues kernel.msgmni = 512 kernel.msgmax = 1048576 kernel.msgmnb = 1048576 # set IPC shared memory blocks, # these are system defaults, # except for shmall which is varies depending on physical memory kernel.shmmni = 4096 kernel.shmmax = 33554432 kernel.shmall = 1073741824 # adjust IPC semaphore settings kernel.sem = 250 256000 64 1024

After changing sysctl.conf you can load the updated values from that file into your running system with "sysctl -p". TCP_WAIT Unlike other systems TCP_WAIT is not an adjustable parameter. For reliable TCP behavior TCP_WAIT should not be lowered below 60 seconds on any system. In the Linux kernel it is hard coded at 60 seconds and would require a recompile to change. To prevent DOS attacks there is a limit on how many sockets can be in a TCP_WAIT state the sysctl name is net.ipv4.tcp_max_tw_buckets its the default value is 180000, with this default one would need to establish, process, and close over 3000 sockets per second to see a problem.

For Solaris
MSGMAP=2048 MSGMAX=65536 MSGMNB=65536 MSGMNI=1024 MSGSEG=32767 MSGTQL=1024 SEMMAP=512 SEMMNI=512 SEMMNU=4096 SEMUME=10 SEMMNS=4096 SEMMSL=8

For HP-UX
dbc_max_pct=10 dbc_min_pct=8 default_disk_ir=1 fs_async=1 max_thread_proc=2048 maxfiles=2048 maxfiles_lim=4096 maxssiz=200802304 (191.5 MB) maxswapchunks=2048 maxuprc=512 maxusers=2000 msgmap=2048 msgmax=65535 msgmnb=65535 msgmni=1024 msgseg=32767 msgtql=2046 semmap=512 semmni=512 semmns=4096 semmnu=4096

For Tru64

ipc:
msg_max = 262144 msg_mnb = 262144 msg_mni = 16384 msg_map = 20000 msg_tql = 4096 shm_max = 4294967295 shm_min = 1 shm_mni = 300 shm_seg = 100 sem_mni = 4096 sem_mnu = 1048 sem_mns = 819200 sem_msl = 1000 sem_opm = 400 sem_ume = 1048 sem_vmx = 32767 sem_aem = 16384 ssm_threshold=8388608 num_of_sems = 1024 max_kernel_ports = 22487

inet:
tcpnodelack = 0 tcbhashnum = 16 tcbhashsize = 8192 ipqmaxlen = 1024 ipqmaxlen = 2048 ipqs = 1 ipqs = 32 tcp_mssdflt = 1460 tcp_msl = 60 pmtu_enabled = 0 tcp_sendspace = 61440 tcp_recvspace = 61440 udp_ttl=60

vfs:
bufcache = 1 fifo_do_adaptive = 0

socket:
somaxconn = 65535 sominconn = 65535

proc:
per_proc_data_size = 3221225472 (3072MB, or three-quarters of the value of per-proc-address-space)

per_proc_address_size = 4294967296 (4096MB)

This is needed if large numbers of Cobol programs are to be compiled. Many applications, for example Oracle and the Tru64 Java Fast VM, need per_proc_data_size to be set correctly. However, this number should not be larger than the physical memory available on the system, otherwise swapping will occur.

Call Telephony Interface (CTI)

RENServer Parameters (psrenconfig.txt)
1. reaper_interval
2. # How often (in sec) to look for and delete expired events. ns_param reaper_interval 600

Default used to be 3600 (1 hour) which caused a steady increase in RENSERVER memory. 3. default_kn_expires
4. # Default to use if an event is received without a kn_expires header. 5. # Use "infinity" or something like "+3600" for 1 hour; must be at least "+5" ns_param default_kn_expires "+10"

Default used to be 3600 (1 hour). Again, this caused a steady increase in RENSERVER memory 6. permission_max_idle
7. # How long (in sec) the permission can stay in cache if the tunnel is closed ns_param permission_max_idle 300

Default used to be 60. This increases the time the PSTOKEN of the agent is expired at the RENSERVER. 8. mtu_size
ns_param mtu_size 1500

New parameter to pad the network TCP packets with additional data to reduce latency in TCP ACK packets.

PSMCAPI Parameters (renclient.properties)

1. interval_topic_reaper
2. # Topic reaper interval in milliseconds

interval_topic_reaper = 3000000

Default value 30000 (30 seconds). This change is to keep the session alive for extended period of time. 3. heartbeats_to_miss
heartbeats_to_miss = 10

New parameter specifies the number of heartbeats to miss before the session is set for deletion. It used to be 2 hard coded in the code. 4. mtu_size
5. #for psmcapi mtu_size=1300

New parameter to pad the network TCP packets with additional data to reduce latency in TCP ACK packets.

PSMCAPI Java Options (StartServer.bat)

1. Java Heap Size Options
-Xms512m -Xmx768m

Default ?Xms256m ?Xmx512m. Increase to above value to handle higher call rate (more than 10 cps)

PSAE and PSAESRV

Much discussion has been made about the importance of PSAESRV versus PSAE. We have conducted a study and shown that PSAE is just as good as PSAESRV for most practical purposes. If you have an AE job that runs longer than 10 sec, PSAE is equivalent to PSAESRV. PSAE has the added advantage of being recycled at the end of each AE job, cleaning up any outstanding SQL cursors to the database that may have been left behind. Since PSAE recycles after each use, it does not has any possible memory leakage problem that may occupied the precious system memory. In short, PSAE is a cleaner workhorse. To shutdown PSAESRV, when you configure the Process Scheduler, you can change the default of the PSAESRV instance to 0.
Values for config section - PSAESRV Max Instances =0 Recycle Count=1000

Allowed Consec Service Failures=2

Reporting Tools
XMLPublisher (8.48 or above)
When using XMLPublisher, the following tips may be useful:

Increase JVM heap size to 512MB. See the "Set Application Server JVM Options" section on how to do this If possible, use PDF template instead of RTF template. PDF template requires less processing and is usually faster than the equivalent RTF template For large XML data sources, PSQuery and rowset data sources may not be the most efficient way to generate the XML file that is used as input to XMLPublisher. You should consider using SQR or other mechanisms to generate the XML file.

Business Objects Enterprise (8.48 or above)

For customers who have large reporting load and have converted the report format to Business Object Enterprise 11 (BOE), to avoid the reporting load affecting other online users, it is recommended that they configure a dedicated application server domain and install the Business Objects Enterprise server on a separate machine. BOE will request data via the integration gateway and Query Access Services (QAS). QAS will make requests to PSANALYTICSRV to retrieve the actual data. The Max Instance Count of PSANALYTICSRV should be set to a value equal to or higher than the maximum number of concurrent reports that need to be run. By default, PSANALYTICSRV recycles after each request. If you are running many small reports that take less than 1 minute to finish, it may be beneficial to set recycle count to a higher value to minimize the cost of restarting PSANALYTICSRV. However, if you are running large reports, the memory footprint of the PSANALYTICSRV may be high and it is better to set recycle count to 0. In addition, if there is a burst of report requests, the PSANALYTICSRV processes spawned to handle the request will continue to consume memory if you set recycle count to greater than 0 (eventually the idle processes will be shutdown). Depending on the size of the document, when generating BOE reports, lots of physical files are created in various layers: BOE temporary files, QAS repository and Report Repository. For better performance these locations should be configured such that the physical files are created on fast disks, optimized for faster read and write While submitting BOE reports thru your Peoplesoft application, the setting "Minutes Before an Idle Connection is closed" need to be set to a smaller value (1 minute) to free up resources from inactive users more quickly. You can change this setting from the BOE Management Console.

Monitoring Tools
PeopleSoft Ping
PeopleSoft Ping is a convenient monitoring facility using PIA and it shows the time spent in different tiers of the PeopleSoft system. It can be accessed from PeopleTools->Utilities>PeopleSoft Ping. Please refer to PeopleBooks for more details.

psadmin
You can use psadmin to monitor Tuxedo queueing. This information can be used to help determine the optimal number of PSAPPSRV processes. See the Application Server Guidelines section for more details.

PeopleSoft Performance Monitor (PPM)

You can capture detailed performance information for individual requests using PPM. It will give you a breakdown of the response time for the request and allow you to see time consumed by individual SQL statements or PeopleCode executions. For more information on PPM, please refer to the Red Paper titled "PeopleSoft Performance Monitor" on Customer Connection.

Change Record
Date Description 07/01/2002 Created document for PeopleTools 8.4 (inherited from 8.1 Performance RedPaper, version 8) 12/20/2002 Revised.

Web Server Guidelines ? Updated service pack information for WebLogicUpdated BEA link, added workaround for Linux. Added section on load balancing Additional Guidelines - Added comment on compression and high-speed link.Rearranged and expanded section on Navigation Pages Caching. Added explanation on session timeout. Kernel Configurations ? Modified file descriptor limit for HP-UXAdded proc parameters for Tru64

01/02/2004 Revised

Application Server Guidelines ? Increased Recycle Count

Date

Description value.Added explanation on PSAPPSRV instance Web Server Guidelines ? Updated Execute Thread Count discussion. Added Jolt Request timing monitoring toolAdded WebSphere Resource Analyzer information Additional Guidelines ? Updated discussion about Compression Web Server Guidelines ? Updated with information about Oracle Application Server Call Telephony Interface ? Added new chapter PSAE and PSAESRV ? Added new chapter

11/04/2005

02/06/2006 Changed formatting 05/23/2006 Added missing step (set auditPwd) to the instructions to enable JOLT request timing trace 10/23/2006 Updated document with 8.48 information

Added information about preload cache, dynamic recycling and application server JVM options Added information on jolt session pooling, weighted load-balancing and dumping heap on OutOfMemoryError Added Integration Broker section Added Reporting Tools section Renamed Kernel Settings section to Operating System Settings section, and added Linux to the section

Oracle Corporation
Author and Date Samuel To, Michael Turner, Michael Simons, Ravi Selvaraj, Indraneel Ganguli, February 2006 Copyright Information Copyright ? 2004, 2005, 2006 Oracle. All rights reserved. Disclaimer This document in any form, software or printed matter, contains proprietary information that is the exclusive property of Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle Software License and Service

Agreement, which has been executed and with which you agree to comply. This document and information contained herein may not be disclosed, copied, reproduced or distributed to anyone outside Oracle without prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any contractual agreement with Oracle or its subsidiaries or affiliates. This document is for informational purposes only and is intended solely to assist you in planning for the implementation and upgrade of the product features described. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described in this document remains at the sole discretion of Oracle. Due to the nature of the product architecture, it may not be possible to safely include all features described in this document without risking significant destabilization of the code.