Oracle Applications on Windows NT

Best Practices from a Support Perspective

Rohit Puri
Oracle Corporation
Introduction
This paper discusses:
How to efficiently implement Oracle Applications (Applications) on Windows NT , primarily
focussing on Release 11. For the purposes of this paper, references to the middle tier include the forms
and the web server. The database tier includes the database, concurrent processing, and the admin server.
Options available to setup fault tolerance in Oracle Applications.
Troubleshooting tips on setting up security in Applications Release 11.
Best Practices in any NT production environment.
How to work effectively with Oracle Applications Support, in addition to describing ways to a fast and
more responsive TAR (Technical Assistance Request) resolution.
Some helpful administration tools from the NT resource kit.
With the ever increasing popularity of Windows NT, Oracle Applications (Applications) can be integrated easily
and quickly into everyday business. To assist companies that need ERP (Enterprise Resource Planning) solutions,
last year Oracle released Applications Release 11.0 for Windows NT. Customers are implementing Oracle
Applications on NT for several reasons:
Low cost of ownership
Ease of administration and maintenance
Strong networking capabilities
High-level security for multiple users
Oracle offers a full suite of Financial and Manufacturing ERP solutions along with the Oracle8 Enterprise
Edition Database on Windows NT. Todays market is strongly moving towards NT solutions. Since the release of
Applications 10.7, there are hundreds of customers that have deployed Oracle Applications on NT. The most
profound change in Applications has been the advent of the internet computing architecture (Figure 1). Oracle has
leveraged this technology to the fullest with Applications Release 10.7NCA and Release 11.
Applications on NT
Architecture and Sizing Guidelines
The key to a successful Applications implementation is a well thought out project plan. With Release 11 of Oracle
Applications, companies can deploy on either single or multiple NT servers. In a single server setup, the database,
tools (Developer 2000, web server, and so on), Applications forms, reports, and library files are all installed on one
NT server, which also acts as the web server for incoming client connections. On the other hand, in a typical multi-
tier environment, the forms and web server are installed on server A, while the database, concurrent processing,
and admin server are installed on server B. Server A acts as the web server for incoming client connections and
connects to the database on server B by using SQL*Net.
During the planning process it is important to consider some sizing guidelines. For the middle tier, each client side
connection requires 10-12 Megabytes of RAM. For example, an NT forms server with two Pentium II CPUs and 1
Gigabyte RAM can support 50 to 60 concurrent users. To add more users, you can enable load-balancing (Metrics
Server) or add multiple forms servers. See Load Balancing and Fault Tolerance. The database tier sizing is more
related to the CPU rather than a limitation of RAM. A database server with four 400Mhz Xeon processors is able
to support 200-300 concurrent user connections. These are rough sizing guidelines based on internal benchmarks
run by Oracle development. The actual number of concurrent users that can be supported depends not only on the
hardware configuration but also largely on what type of processing jobs are being submitted, such as batch jobs,
data entry, idle time, and so on.
Figure 1.
Windows NT Registry and Environment Variables
Most Oracle Applications environment settings are stored in the Windows Registry. You dont have to reboot NT
for the changes made in the Registry to take effect. Application processes on NT are coded to look at environment
or Registry values upon startup. This means that you only have to bounce the Oracle Application service or process
(e.g., Concurrent Manager Service) for the environment changes to take place. You can set up environment
settings in different places. Windows NT searches for environment variables in the following order:
1) Command Shell (DOS Window). These settings will only affect programs started from that window (e.g.,
Developer/2000 Forms server)
2) In the Control Panel, under the Environment tab under System.
3) Variables stored in the Registry.
Note: Keep the above order in mind when troubleshooting.
Oracle Applications Release 11 on Windows NT also supports the concept of multiple Applications TOP
directories, which allows for multiple Applications installs on the same server. Each installation of Applications
creates a separate environment hive in the NT Registry under HKEY_LOCAL_MACHINE -> SOFTWARE ->
ORACLE -> Applications. To change which applications installation to use, set up the APPL_CONFIG
environment variable that is stored under each of the above hives.
Note: It is recommended that you only implement multiple APPL_TOPs in non-production environments
(e.g., in a test or training database). For production installations, Oracle recommends running
Applications on a machine that is configured with a single APPL_TOP.
Load Balancing and Fault Tolerance
To eliminate a single point of failure in an environment with more than 50-60 concurrent users, it is better to
spread the client load to multiple middle tiers. By enabling load-balancing, you can set up Oracle Applications to
reduce a single point of failure for the middle tier.
To eliminate a single point of failure in the middle tier, you can set up multiple Metrics Servers (A & B) in the
event one server fails due to hardware, OS, or process failure. For this purpose, each forms server machine should
have two Metrics client services running: one reporting to Metric Server A and the other reporting to Metric Server
B.
You can customize the names for the Metrics Client service name with Forms 4.5.10.8. When no argument is
passed to the D2LC install program, the default name D2LC Client Service is used for the Metrics Client
service name; otherwise, it takes the user defined argument. For example, to install three Metrics clients on the
same machine with different names, you need to explicitly create three services:
1. d2lc1032.exe install
-D2LC Client Service installed.
2. d2lc1032.exe install MetricsClientA
-MetricsClientA installed.
3. d2lc1032.exe install d2lc client 2
-d2lc client 2 installed.
Note: Use double quotation marks for names with punctuation or spaces.
While this procedure can satisfy Metrics Server elimination as a single point of failure for the middle tier, there
are different things to try after a Metrics Server goes down. The following three setups are common:
1. One web server and web request broker. Users connect to two Metrics Servers via two separate
cartridges set up on the web server. Users will have to manually reconnect by using the second cartridge
when one server is down.
2. Two web servers that use two URL access paths: Users have to manually reconnect by using the second
URL when one server is down.
3. Two web servers that use a single virtual URL by using a third-party fault tolerance device (ex. Ciscos
Local Director, HolonTechs HyperFlow.)
Oracle is currently looking at the possibility of certifying Applications on Windows NT to work together with
Microsoft Cluster Server. This setup will use Oracles fail-safe technology to eliminate a single point of failure
in the database tier.
Since all executable and library files are prelinked on Windows NT, there are no relinking problems. Most load-
balancing problems are usually related to the setup. To set up load-balancing, see Deploying Applications on the
Web Developer 2000 Manual, Chapter 2 (Part # A57514-03).
Administering Oracle Applications Security
A security feature must be set up after an Applications installation to use the functionality of Workflow and Self-
Service Web Applications (SSWA). Because Release 11 is deployed in a multi-tier configuration, the security
model has been enhanced to include authentication of application servers to the database servers they access. When
this layer of security is activated, the application servers pass server IDs to the database server. If the database
server recognizes the server ID, it grants access to the database. You can create server IDs by using a Java script
and you must enable security after an Applications install. The Java script, AdminAppServer, is used to set up,
activate, and check the status of the application server security feature. See the following steps and troubleshooting
tips to set up security and SSWA:
1. Set up the Java Cartridge specific parameters by adding the %ORACLE_HOME%\jdbc\lib\classes102.zip
to the end of the default setting for CLASSPATH.
2. Add a new row defining the following: SYSTEM_PROPERTY=APPS_OCI_LEVEL=7
Note: To configure the above two steps, navigate to Cartridge Configuration -> Java Cartridge -> JAVA
Cartridge specific parameters screen from your admin page of the web server.
3. Navigate to the Cartridge Summary (WRB) screen, scroll to the virtual directories, and add a new virtual
directory:
/OA_JAVA_SERV JAVA %OH%forms45/java
Note: Point the path to a valid directory where the java class files are installed.
4. Set the Java cartridge to use the process model. In the Cartridge Summary (WRB) screen, under the
Applications & Objects section, make sure that the thread (T/P) entry for the Java Cartridge is set to P
instead of T.
5. Create DBC files (text files that contain database connect information) for each database that the
application server will connect to. Use the AdminAppServer script to generate these files:
java oracle.apps.fnd.security.AdminAppServer apps/apps@dbname \
ADD GWYUID = APPLSYSPUB/PUB FNDNAM = APPS \
GUEST_USER_PWD = VISITOR/VISITOR
6. For authentication to work properly, add the following parameter to your initial html file on the middle
tier. If you do not, the following error message appears: App-09999: Application server does not have
permission to access the database. In the serverArgs parameter, add:
Example of default ServerArgs parameter:
param name=serverargs value="module= /apps/fnd/11.0.28/forms/US/FNDSCSGN.
At the end of this parameter, add CONFIG=appsdb_apps using the name of your .dbc file
7. By default, server security is not turned on. To turn on authentication, enter the following statement:
java oracle.apps.fnd.security.AdminAppserver apps/apps@dbname AUTHENTICATION ON
8. Before starting your Forms server, manually copy the APPL_SERVER_ID parameter from the .dbc file
and set it as an environment variable in a DOS window or in the System tab under Control Panel. Now,
start the Forms Listener in that window.
Troubleshooting Tips :
Make sure that a file named dbhostname_twotask.dbc is created in the $FND_TOP/secure directory after
running the above Java script AdminAppServer.
Create an applications user, VISITOR with no assigned responsibility. This userID will be used by the
GUEST_USER_PWD parameter in the example above (see step 5.)
If you get this error; Thread Null when creating the .dbc file, make sure that you are using the correct
JDK (Java Developers Kit) and that the \JDK\BIN\ directory is present in the path environment setting.
When trying to create the DBC file, if you get errors indicating that the script cannot find certain Oracle8
DLL files (only in multi-tier environment), install Net80 in the middle tier since java.exe requires
references to certain Oracle8 library files.
When logging into Applications, you get a dialog box with an APP-1630 error after you turn on
authentication, the newly created .dbc file is not being read by the Forms server. Make sure you followed
step 8 in Administering Oracle Applications Security.
Oracle Alerts: MAPI Compliant
The Oracle Alert module in Applications Release 11 for Windows NT provides direct support for MAPI
(Messaging Application Program Interface) compliant mail systems. This feature enables Alerts to send and
receive messages, and perform response processing by using a MAPI compliant server like Oracle InterOffice 4.1
or Microsoft Exchange. See Appendix D in the Windows NT Installation manual for Oracle Applications (Part#
A57978-01).
Jinitiator
Jinitiator is an add-on to a browser that allows Oracle Applications to be brought up through a browser. Jinitiator
is currently under testing and will be released soon.
Patching Strategy
The code in Oracle Applications is periodically updated. There are two types of updates:
1. Maintenance Packs: Beginning with Release 11.0, Oracle Applications will release regular maintenance
packs. A maintenance pack contains cumulative patches for each product. You can install product mini-
packs individually, or install a consolidated patch. Each maintenance pack is cumulative. All the fixes
made in 11.0.1 are included in 11.0.2. 11.0.3 will include all of the fixes made in 11.0.1 and 11.0.2.
2. Mini-Packs: Product mini-packs exist for each individual Oracle Applications product (e.g., AP, AR, GL,
and so on). You can apply these patches for individual products.
Always test updates on a development or test instance prior to installing updates on a production system.
Note: Oracle does not support a method for removing patches from the application or database server, so it is
critical you first test them before applying them to your production system.
On Windows NT, there are two types of processors; Dec Alpha NT and Intel NT. When you get a patch, make sure
it is for the correct hardware architecture. The same patch may exist for SmartClient 16.1, NCA (10.7), or Release
11, so make sure that you get a patch for the correct version of Applications.
A patch may consist of many files (patch driver files, readme, replacement files, scripts, and so on) and may need
to be run on either the middle tier, the database tier or both. When applying a patch, follow these guidelines
outlined below:
Download the most current version of the patch from Oracle Support Services (OSS.)
Copy the patch to a common location for patches on your system.
Review the readme that comes with the patch to determine which tiers to apply the patch.
If there is an Autopatch (adpatch.exe) that comes with the patch, use it to install the patch.
Before applying a patch, be sure to back up everything.
Make sure all Oracle Applications users log out of the system.
Shutdown the concurrent manager service if patch requires (optional).
Perform any steps that are listed in the readme.txt file before running Autopatch.
Run Autopatch and check the log files for errors (adpatch.log).
To check which patches have been applied by using adpatch.exe, view the applptch.txt file which
maintains a record of all applied patches.
The patching technology for Release 11 has changed so that when a new form is provided as a part of a bug fix, it
is regenerated onsite. This preserves customizations made to all dependent forms. If you are running into a
problem and cannot find a solution on MetaLink, call Oracle Support Services (OSS) to obtain a resolution by
opening a TAR (Technical Assistance Request.) If OSS determines that it is a new problem, they will reproduce it
inhouse and work with the developers to fix it.
Maintain a log of all patches applied on both the database and middle tiers. OSS will need to know if certain
patches were applied and such an audit trail ensures a quick solution to your problem. Keep all the log files from a
patch in a separate archive along with a copy of the patch in case you need to go back to check or reapply the
patch.
Newer versions of the database or tools (Developer 2000, Web Server) require InterOperability patches that are
provided on an InterOperability CD. With the latest release of the InterOperability CD, Oracle now includes
patches for different database versions on a single CD. Make sure that you apply the patches from the
InterpOperability CD for the correct version your database.
Applications on Windows NT: Best Practices
1. Before calling Oracle Support Services, make sure that you check some basic setup configurations:
You can connect to the database tier from the middle tier by using sql*plus or tnsping.
Bring up the admin and www listener web pages on the web server.
You started the Forms server on the correct port (Check task manager for f45srv32.exe and tlist.exe to
check the port on which the forms server was started).
Make sure that the settings for ORACLE_HOME and LOCAL are correct.
2. If you are using a single machine to home multiple APPL_TOPs, keep separate applmgr accounts to
manage each APPL_TOP. Each NT account stores different settings for the user and eliminates confusion.
3. If you are running multiple forms servers on a single machine, use tlist.exe from the NT resource kit to
differentiate between the instances the f45srv32.exe process connects to.
4. Maintain different log files under the APPL_TOP. Regularly archive and purge files that are not needed to
save valuable space.
5. Since the Concurrent Manager Service depends on the database to be already up and running, make sure
that you specify the [DEPENDENT_SERVICE] parameter when creating the service by using
cmsrvadm.exe.
6. To start or stop the concurrent manager service manually from the command line, enter the SC.EXE
command from the NT resource kit rather than from NET START/STOP. The Service Control (SC.EXE)
program does not time out and is more efficient than NET START/STOP.
7. NT has a robust file security system, USE IT. Make sure no one is allowed access to the file system other
than authorized users.
8. After an installation is complete and tested for the production environment, remember to change all the
default passwords. This includes the SYS and SYSTEM schemas, the applications base schemas like GL,
FND, AP as well as Oracle Applications users like SYSADMIN
Caution: You need to follow a specific method to make changes to the Applications database schema
passwords. Please download the steps from MetaLink or call OSS for detailed instructions.
9. Monitor database space usage. If you are not using any of Oracles Enterprise Manager tools to do this,
use the built-in Applications Alerts.
10. To function correctly, every version of Oracle Applications requires you to use a specific combination of
the operating system, database, and tools like Developer 2000, web server, and so on. It is critical that you
only use Oracle certified combinations. Oracle does not guarantee support for non-certified versions of the
database or tools. For a detailed certification matrix, check Certify under MetaLink or call OSS for a copy.
11. Document any customizations made to the system. Customizations are not supported by Oracle Support.
Maintain an instance with no customizations for testing purposes.
12. Before upgrading an existing installation, read the release notes and patch readme files. Always perform a
test upgrade before making any changes to a production environment.
13. Document and test a complete backup strategy. Make sure you are prepared if you need to recover after a
failure (hardware crash, software or database corruption). This is true for all tiers.
Oracle Support Services (OSS)
There are several options available to customers when they need assistance with any Oracle product. This section
describes how customers can work effectively with OSS (Oracle Support Services) and how to get optimal service
and quick TAR (Technical Assistance Request) turnaround.
Support Options available to customers
Oracle MetaLink is Oracle Support Services premier web support service that is available to OracleMetals (Gold,
Silver, Bronze) customers, 24 hours a day, seven days a week. It is a quick and convenient alternative to calling
OSS. MetaLink provides information within several areas of interest:
MetaLinks Technical Libraries include product-specific alerts, de-support notices, latest product versions,
top product articles and bulletins that are at your fingertips. You can also research an upgrade path at your
convenience.
Installation Assistance and Product Reference Area: You can find step-by-step installation instructions,
installation manuals, release notes, readme files, product documentation, technical support bulletins, and
white papers.
Technical Support Knowledge Base: This is a repository of information with a powerful search engine
that enables full-text queries from all published data in technical support bulletins, problem and solution
articles, technical alerts, readme files, white papers, release notes, installation instructions, and technical
forums. The knowledge base search is available at the top of every page within MetaLink.
Technical Forums: These online forums provide an opportunity to communicate with Oracles users, find
information, and post questions. Oracles Technical Analysts monitor the forums daily and respond to
technical questions within two business days.
Certify is another web-based tool that provides up-to-the-minute interoperability information for Oracle
Applications product suites. You can use Certify to identify problems that are discovered after a products
release, and save considerable time and effort diagnosing and fixing problems. Certify provides lists of
certified combinations of operating system, database, Applications, and tools versions. Brief as well as
detailed listings of recommended or required patches for individual products are available. Soon, Certify
will cover Oracles other major product suites, in addition to the current Applications products, which
includes Financial Services, Industry Applications, and Internet Products.
iTars: MetaLink will soon offer Oracle customers the ability to manage their TARs over the World Wide
Web. An iTAR is a TAR that has been submitted by a user through MetaLink (Figure 2.). You can choose
to be contacted by either MetaLink, FAX, Email, or phone when logging an iTar. Once submitted, the
iTAR is handled just like any other TAR. You can create, read, update, and close TARs through
MetaLink and generate customized TAR management reports. With internet TARs integrated into the
standard TAR process of OSS, you have a consistent service for all TARs logged globally and hold times
are further reduced.
Oracles Fax-on-Demand service is a time-saving, easy-to-use alternative to contacting Support Services.
Requested information is faxed within 24 hours. By using Fax-on-Demand, you can:
Receive copies of TARs created with a CSI Number.
Retrieve records from Oracles bug database.
Access Support bulletins, product availability information, and Y2K documentation.
Retrieve phone menu routing sheets.
Before dialing the Fax-on-Demand number, make sure you have your CSI number, fax number, and correct
platform code ready. See the http://www.oracle.com/support web site for more information.
Figure 2.
Active TAR management
Oracle Applications are complex to manage and consist of many different integrated software modules; therefore
you can run into many problems. You can contact Oracle Support Services (OSS) to log TARs to resolve problems.
It is in your own interest to actively manage and prioritize problems with OSS. Routinely, collect and communicate
all problems to OSS. When there is an urgent need for a solution, communicate critical timelines at an early stage
and provide a business case for the urgency to escalate a TAR. Personnel managing issues should identify
themselves and prioritize TARs to focus on key issues. Adopt a TAR logging and tracking methodology. Track all
ongoing as well as previously closed issues with OSS.
When a problem arises that is not reproducible in Oracle Supports test environment, an analyst may dial-in to the
Windows NT desktop at the customers site by using PcAnywhere by Symantec .
Oracle Support Services: Best Practices
Before logging a TAR, research a problem and its symptoms vigorously on MetaLink to avoid spending a lot of
time on the telephone with OSS. When calling OSS, describe your issue in detail. Before calling, make sure you
have the following information:
CSI #, contact name (two contacts), phone #, alternate phone #, fax #, e-mail.
Operating System of all tiers (client, forms server, database server) involved in the installation.
Versions of the operating system with level of service packs (Intel NT vs. Dec Alpha NT).
Oracle Applications, database, and if applicable, the tools versions.
Business impact to the company to determine the severity of the issue.
List of all related errors from Application log files.
Module names and versions for the forms or reports that are erroring out.
List of any changes made to the system, including any patches recently applied. These can be operating
system, Oracle Applications or Database patches.
Steps to duplicate the problem.
Dial-in information for severity 1 TARs, including the username, password, and dial-in timings.
24-hour contact information, including pager # or mobile telephone # for severity 1 TARs.
Customizations made to the system.
NT Resource Kit and System Tools
The Windows NT Resource Kit provides many tools for the system administrator. The following tools can be used
to manage Oracle Applications on Windows NT.
srvany.exe allows users to run any Windows NT application as a service. Srvany.exe allows any application to run
within a logon account that is different from the one where the application was started from. The application keeps
running and services requests even when no one is logged on to the server. The Developer/2000 Forms Listener is
a good example of turning an application into a service.
ipconfig.exe shows valuable information about a computers network settings including TCP/IP address,
hostname, DNS server, WINS server, and other network parameters. This information is helpful when
troubleshooting SQL*Net problems.
sc.exe (Service Control) is useful for developers of services since it provides more detailed and accurate
information about services than the two end-user utilities provided with the Windows NT operating system (NET
START/STOP). The Services Control Panel applet and the network command-line interface (NET.EXE) can
determine if a service is running, has stopped, or has paused. Although these tools are fine for some applications,
the information they provide about a service can be misleading. The Concurrent Manager service is a good
example of where to use SC.EXE to control the service. To stop or start a Concurrent Manager service on
Windows NT takes longer than the timeout built into NET START/STOP. This timeout can lead to erroneous
messages regarding the status of a service; hence, it is important to use SC.EXE.
tlist.exe (Task List Viewer) is a command-line utility that displays a list of tasks or processes currently running on
the local computer. For each process, this utility shows the process ID number, process name, and if the process
has a window, the title of that window, and any parameters that were passed when the task was started. If you are
running multiple forms server listeners on a single NT machine, this tool is invaluable to check which TCP/IP port
a forms server was started with.
pmon.exe (Process Monitor) is a command-line tool that monitors process resource usage and tracks CPU and
memory usage. Use it to troubleshoot memory and performance problems.
pviewer.exe (Process Viewer) is a Windows-based tool that displays information about a running process and
allows you to stop processes. The memory tab under Process Viewer provides more details on how much memory is
allocated to each process.
Performance Monitor is a GUI tool built into the Task Manager and provides detailed resource utilization data for
all processes running on the system. Oracle has integrated with Performance Monitor, so that you can view
utilization of operating system and Oracle resources (e.g., file write bytes per second related to Oracle, library
cache hit ratios, and so on).
Event Viewer (GUI Administrative Tool) allows users to troubleshoot various hardware and software problems. It
is a tool that keeps a log of any system, application, or security event. The Event Viewer utility allows the
administrator to view system alert messages on NT. Oracle has integrated with Event Viewer such that Oracle
startup and shutdown messages and the OS audit trail (if you configure OS auditing in Oracle) appear there.
Conclusion
With the market trend strongly supporting Applications, Windows NT certainly provides a cost-effective, robust,
and flexible solution for implementing Oracle Applications. Oracle is dedicated to providing the most current and
robust ERP solutions on Windows NT. Through careful planning you can help to ensure a successful Oracle
Applications Installation. Following the guidelines outlined in this paper can reduce the time, effort, and
frustration involved.
References
Oracle Internal Web Pages (Only available to Oracle employees)
appsweb.us.oracle.com
Global Sales Practices Oracle Applications
ntssnet.us.oracle.com
NT Sales Solution
agsl.us.oracle.com
Applications Global Service Line
World Wide Web Pages
www.oracle.com/support
Oracle Support Services Home Page
www.indirect.com/www/ceridgac/ntsite.html
MS Windows NT Sites (A-Z)
www.microsoft.com/ntserver
Microsoft Windows NT Server page
www.registrysecrets.com
Secrets for Windows NT Registry
www.oracleassist.com
OracleAssist Oracle Discussion Forums, Jobs, Links, News and More
www.WindowsNTFans.com
The Windows NT Fan Club
www.oraclefans.com
The Oracle Fan Club for Oracle Users
About the Presenter
Rohit Puri is a Senior Technical Analyst with Oracle Support Services. He works for the Applications installation
and AOL (Application Object library) group which is involved in providing technical support to clients that are
installing, maintaining, and troubleshooting Oracle Applications.