4/21/2011 1011248.

6
Oracle Corporate Support
Problem Repository

1. Prob# 1011248.6 WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

2. Soln# 2059281.6 MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

1. Prob# 1011248.6 WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

Problem ID : 1011248.6
Affected Platforms : Generic: not platform specific
Affected Products : Oracle Application Object Library
Affected Components : FOUNDATION V10.XX
Affected Oracle Vsn : Generic

Summary:
WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

+=+

Problem Description:
====================

What are the common concurrent manager issue in applications release 10?

Key words: DIAGNOSIS EXAMPLE ORACLE USDSOP PERMISSIONS LOG OUT APPLCSF CHILD
PROCESS AFPEOT REP_PATH APPLPLS APPLSYS.ENV ORACLE_SID GWYUID FNDNAM TWO_TASK
AFPCAL CPMGR AFPCSQ AFPGCP SYS SYSTEM DUAL AFPGMG PMON LOCK
FND_CONCURRENT_REQUESTS STARTMGR AFPCMT AFPSMG AFPGSR V$PROCESS V$SESSION
V$PARAMETER DBMS_LOCK DBMS_PIPE STANDARD FND_V$PROCESS FND_CONCURRENT
AFPFPROS.PLS AFPFPROB.PLS AFCPDCPS.PLS AFCPDCPB.PLS AFCPUTLS.PLS AFCPUTLB.PLS
FNDCPMGR APPLDCP AFPPRT FDXLNG NLS_LANGUAGE

+==+

Diagnostics and References:

* {7022.6,Y,100} RELEASE 10 OF APPLICATIONS

2. Soln# 2059281.6 MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

Solution ID : 2059281.6
For Problem : 1011248.6
Affected Platforms : Generic: not platform specific
Affected Products : Oracle Application Object Library
Affected Components : FOUNDATION V10.XX
Affected Oracle Vsn : Generic

Summary:
MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

+=+

Solution Description:

C:/…/1011248.6 1/10
4/21/2011 1011248.6
=====================

Common Concurrent Manager Issues in Release 10

The concurrent manager is the heart of background processing within

Oracle Applications, and is by no means trivial to control or
diagnose. This bulletin attempts to pull together common problems
encountered by the concurrent manager. It is arranged first by error
number, and then follows with a list of more general solutions to
problems.

Error: APP-00902 usdsop cannot redirect standard output

1. Make certain that the OS login for the user trying to start
the concurrent manager is the owner of the applications code, or
has been given the appropriate permission to run startmgr and
batchmgr. This can be done by a) making the startmgr script
setuid for applmgr, b) changing the group permissions on the
scripts and placing users any users in that group who should be
able to bring up the concurrent manager.

2. Check to see that the permissions are correct on all of the

log directories.

a) If $APPLCSF is set so that log and out files are written

to a common directory, then make sure the user can write to
$APPLCSF/$APPLLOG.

b) Check the permissions on $FND_TOP/$APPLLOG. This should

be done whether or not $APPLCSF is set because the internal
manager log file will always be written to
$FND_TOP/$APPLLOG.

c) If $APPLCSF is not set, check the permissions on all of

the $<prod>_TOP/$APPLLOG directories. Note: Although the
log directories are not used when $APPLCSF is set, they must
still exist under each product.

3. Check that the value of $APPLLOG is less than 8 characters in

length.

Error: APP-00912 Child process XXXXX exited with status 0. Manager

not activated.

The internal concurrent manager will only start up those managers

which have a control code of A (for Activate) or R (for restart).
Login to the applications with the System Administrator
responsibility and follow the menu path: \Navigate Concurrent
Manager Administer. For each manager which needs to be started,
tab over to the Control column and select Activate.

You will still see the above message for any managers not

C:/…/1011248.6 2/10
4/21/2011 1011248.6
activated and, although listed in the logfile, it is not a
critical error. Exiting with status 0 is an indication that a
program exited cleanly. See chapter 8 of the System
Administration manual for more information.

Error: APP-00988 ORACLE error ecountered.

This is a generic error and precedes an Oracle error. See the

section on ORACLE errors for specific examples.

Error: APP-01080 afpeot: Cannot initialize concurrent request

information for request ID XXXX.

1. Check that APPLLOG, APPLOUT, and if set, APPLCSF, are pointing

to valid directories. Shutdown the concurrent manager, start
with a clean environment (it may be easiest to logout and back in
again, and rerun the APPLSYS.env file), and bring the concurrent
manager up again.

2. The environment variable REP_PATH should only include

$AU_TOP/$APPLPLS. In previous releases it pointed to other
directories, and by default when the APPLSYS.env file is created
it sets REP_PATH to also append previous values of the variable
to itself (REP_PATH=$AU_TOP/$APPLPLS:$REP_PATH). Therefore, if
the environment file is read several times before starting the
concurrent manager, REP_PATH grows to be longer than a single
directory path, even though it is the same directory being listed
multiple times.

Error: APP-01086 Can not set profile option value

Check that the correct $ORACLE_SID, or $TWO_TASK, is being

pointed to, and also check that $FNDNAM and $GWYUID are correct.

Error: APP-01114 AFPCAL received failure code while running CPMGR

This is a generic error message which is usually at the end of a

list of several applications errors. These include but are not
limited to:

1. APP-01227 XXXXXX encountered ORA-XXXX

a. AFPCMT encountered ora-3114: not connected to oracle

Check to make sure the database is actually up, and that

the correct ORACLE_SID is being used.

b. APP-01227 AFPCSQ encountered ORA-1005: null password

Check to see that the database is up and the values are

correct for ORACLE_SID, FNDNAM, and GWYUID. Check the
PMON method being used by the concurrent manager. Often
using the RDBMS method will avoid these errors.

C:/…/1011248.6 3/10
4/21/2011 1011248.6

c. APP-01227 AFPGCP encountered oracle error SQL-02112

select into returns too many rows

Check that there is only one row in sys.dual and

system.dual.

d. APP-01227 AFPGMG encountered oracle error ORA-00923:

FROM keyword not found where expected

i. Check the permission on the $APPLLOG directories.

Note that they need to be correct for all the $APPLLOG
directories, and not just under $FND_TOP.

ii. If the sql statement being executed involves

recursive SQL, this could be base bug 246213.

iv. Check the pmon method being used, and try using the
LOCK pmon method instead.

2. If there is no log file at all, but this error message

appears when trying to start the concurrent manager, check
the following:

a. The permissions are correct on all the $APPLLOG

directories, and they actually exist. This includes
each of the $<prod>_TOP directories, and not just
FND_TOP.

b. There aren't any terminated requests in the table

FND_CONCURRENT_REQUESTS. Execute the following SQL
statement as the applsys user:

select count(*)
from fnd_concurrent_requests
where status_code='T';

c. Although the startmgr script is supposed to clean out

any requests that have a status of 'Terminating', it
occasionally balks before doing so. The solution is to
either remove these requests from the table:

delete from fnd_concurrent_requests

where status_code='T';

or change the status to 'Terminated':

update fnd_concurrent_requests
set status_code='X', phase_code='C'
where status_code='T';

3. APP-01227 AFPCMT encountered an ORACLE error. ORA-01012:

not logged on
APP-01227 AFPSMG encountered an ORACLE error. ORA-01012:
not logged on

C:/…/1011248.6 4/10
4/21/2011 1011248.6
APP-01227 AFPGSR encountered an ORACLE error. ORA-03113:
end-of-file on communication channel

Most likely the database was shutdown or crashed while

the concurrent manager was still running. Shutdown the
concurrent managers, bring up the database, and then
bring the concurrent managers back up.

Error: APP-01125 Concurrent manager encountered an error while

running XXXX

Check that the executable XXXX actually exists, and is in the

correct location. Attempt to run the executable from the command
line and see if the problem persists. Specifically compare the
arguments passed on the command line and the arguments used by
the concurrent request (this should be in the request logfile).
Finally, try relinking the executable.

Error: APP-01144 Error while finding the status of the concurrent

manager.

1. You may have a corrupt or missing view, or be missing some

grants and/or synonyms. Check that the applsys user has access
to: V$PROCESS, V$SESSION, DBMS_LOCK, DBMS_PIPE, STANDARD. Other
Applications oracle ids also need access to these views and
packages.

If the view is in fact missing or invalid, rebuild it. From the

operating system, as the applmgr user type:

FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN

'System Administrator' SYSADMIN

or as the applsys user run the following SQL statement

Create View Fnd_V$Process as

( Select P.Addr, P.Pid, S.Process Spid,
S.Osuser Username, P.Serial#,
S.Terminal, S.Program, P.Background,
P.Latchwait, S.Lockwait
From V$process P, V$session S
Where P.Addr = S.Paddr );

2. As the applsys user run the following SQL statement:

select object_name from user_objects where status='INVALID';

If any of the FND_CONCURRENT% objects are invalid, run the

following PL/SQL scripts (from $FND_TOP/install/sql) as the
applsys user:

AFPFPROS.pls
AFPFPROB.pls

C:/…/1011248.6 5/10
4/21/2011 1011248.6
AFCPDCPS.pls
AFCPDCPB.pls
AFCPUTLS.pls
AFCPUTLB.pls

NOTE: Run the scripts in the above order.

Error: APP-01167 The internal concurrent manager encountered an

error.

This is another generic error which may be followed by Oracle

errors, or:

APP-01054 Distributed concurrent manager enabled, lock

PMON method requested
APP-01114 AFPCAL received failure code while running
FNDCPMGR

Make certain that $APPLDCP is set to off or, if using

distributed concurrent processing, change the PMON
method to LOCK by running afimpmon.sql from
$FND_TOP/sql.

Error: APP-01266 in AFPPRT

Check that there aren't any requests with a status code of

'Terminating' in the table fnd_concurrent_requests by executing
the following SQL statement as the applsys user:

select count(*)
from fnd_concurrent_requests
where status_code='T';

If this returns anything other than 0, then run:

delete from fnd_concurrent_requests

where status_code='T';

OR

update fnd_concurrent_requests
set status_code='X', phase_code='C'
where status_code='T';

Oracle Errors
-------------

Error: ORA-12705 in FDXLNG

This occurs because the value of nls_language is NULL in the

table FND_CONCURRENT_REQUESTS. To fix this make certain that the
environment variable $APPLLNG is set and correct (usaeng in most
cases), and also execute the following SQL statement as the
applsys user:

C:/…/1011248.6 6/10
4/21/2011 1011248.6

Update Fnd_Concurrent_Requests
Set Nls_Language = Substr(Userenv('Language'),1,
Instr(Userenv('LANGUAGE'),'_')-1),
Nls_Territory = Substr(Userenv('Language'),
Instr(Userenv('LANGUAGE'),'_')+1,
Instr(Userenv('LANGUAGE'),'.')-
Instr(Userenv('LANGUAGE'),'_')-1)
Where Nls_Language is null
or Nls_Territory is NULL;

ORA-942 Table or view does not exist (when starting up the ICM)

1. Check that the APPLDCP is set to off, unless parallel

concurrent processing is being used. Specifically DBMS_LOCK
needs to be accessible by the applsys user when APPLDCP is ON.

2. Check that the view FND_V$PROCESS exists, and that the applsys
user has access to V$PROCESS, V$SESSION, DBMS_LOCK, DBMS_PIPE,
STANDARD, and, if using distributed concurrent processing,
V$THREAD.

Generic Issues
--------------

1. The concurrent manager continuously spawns new processes although

there is no obvious activity in terms of concurrent requests.

Change the PMON method to OS. This can be done by running

afimpmon.sql from $FND_TOP/sql as the APPLSYS user. Be sure to
shut down the concurrent manager before running this script.
WARNING: Using OS PMON method is not support in 10.4.

2. The concurrent manager starts from the command line, but not from
the Administer Concurrent Manager screen.

a. This is generally a problem with permissions on the

$FND_TOP/$APPLLOG directory. Check to make sure that its
writable by the user that started the concurrent manager, and
even more importantly make sure the directory is not a symbolic
link. When starting the manager from within Applications, it
incorrectly verifies that this directory exists, but a symbolic
link returns the wrong response to the query.

b. As the applsys user run the following SQL statement:

select object_name from user_objects where status='INVALID';

If any of the FND_CONCURRENT% objects are invalid, run the

following PL/SQL scripts (from $FND_TOP/sql) as the applsys user:

AFPFPROS.pls
AFPFPROB.pls

C:/…/1011248.6 7/10
4/21/2011 1011248.6
AFCPDCPS.pls
AFCPDCPB.pls
AFCPUTLS.pls
AFCPUTLB.pls

NOTE: Run the scripts in the above order.

3. When starting the concurrent manager FNDLIBR dumps core.

Try relinking FNDLIBR, if that does not check to see if there are
pending requests. If possible, make certain there aren't any
jobs in pending status, and then try to bring the manager up.
This is especially true if there are a high number of requests
with a status of pending because the concurrent manager reads in
all pending requests on startup.

To delete pending requests execute the following statement as

APPLSYS user:

4. The specialization rules that were just recently defined are not
taking effect.

Try bouncing the CM to make sure that all the current rules are
being implemented.

5. On VMS starting the manager results in an error indicating that the

user should verify they are logged in as the applmgr user, when in
fact the user already is.

Make sure that the queue for concurrent requests exists. By

default this queue is called FND$STDMGR.

6. When navigating to the concurrent requests screen (\Navigate

Concurrent Requests from the System Administrator Responsibility) it
justs sits there and hangs.

This is most likely due to base bug 246213. Make certain this
patch has been applied. It is actually a 7.1.3 RDBMS bug which
affects complicated views.

7. Although $APPLCSF is set, the internal concurrent manager log is

being written to $FND_TOP/$APPLLOG.

This has been logged as bug 259415. At this time there is no

patch for this, but it can be modified by changing the startmgr
(or startmgr.com) script to use $APPLCSF instead of $FND_TOP when
creating the logfile.

8. All the concurrent managers are running, and the concurrent manager
has been successfully bounced several time, but all the requests have
a status of 'Inactive' and 'No Manager'.

Check that there aren't any pending requests at all. If an

run-alone immediate concurrent program is submitted, but the
program is not registered in any of the concurrent libraries then
the status will be set to 'Inactive' and 'No Manager' because

C:/…/1011248.6 8/10
4/21/2011 1011248.6
none of the managers could run it. Any requests that follow this
request will then be unable to run until the original request
completes.

9. The concurrent manager forms within application indicate that the

concurrent manager is down, when in fact it is running and processing
requests without any problems.

This is a bug when using the LOCK pmon method. It is fixed in

10.4.1 and the workaround is to use RDBMS or OS pmon method.

10. Terminated concurrent processes that are "spawned" do not die,

although the View Concurrent Requests screen says that have been
Terminated. Also, after some time the Concurrent Requests screen will
changed the status to 'Completed Normal'.

This is bug 249158, and is known to occur on at least HP. A

patch is available.

11. How to recreate the view FND_V$PROCESS:

FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN 'System Administrator'

SYSADMIN

or as the applsys user execute the following SQL statement:

Create View Fnd_V$Process as

( Select P.Addr, P.Pid, S.Process Spid,
S.Osuser Username, P.Serial#,
S.Terminal, S.Program, P.Background,
P.Latchwait, S.Lockwait
From V$process P, V$session S
Where P.Addr = S.Paddr );

12. If there are multiple internal concurrent managers in the same

database, only one of those managers can use the LOCK pmon method.

13. When using parallel concurrent processing, the concurrent manager

names can not have embedded spaces in them. The workaround is to
replace the space with an underscore, or change the name to a single
word.

------------------------------------------------
When in doubt....

If you are encountering errors with the concurrent managers some steps
that can be safely executed include:

1. Bounce the concurrent managers. Bring them down, and then

making sure the environment is properly set up bring them back up
again. Remember that the concurrent managers' environment is set
at the time they start up, and any modifications that you make to
the environment after that will not be picked up until the
concurrent managers are bounced again.

2. Check for the existence of the $APPLLOG directory under each

PRODUCT_TOP directory, and that the permissions allow for writing

C:/…/1011248.6 9/10
4/21/2011 1011248.6
to those directories. If $APPLCSF is set, so that all the log
and out files get written to a common directory, the $APPLLOG
directories still must exist. Also, even with $APPLCSF set, the
internal concurrent manager log files will always be written to
$FND_TOP/$APPLLOG.

3. Make certain the login id that is starting the concurrent

manager has permission to do so. By default just applmgr can do
this.

4. Check that there aren't any requests which have a status of

'Terminating' in the table fnd_concurrent requests:

select count(*)
from fnd_concurrent_requests
where status_code='T';

If there are any such rows then delete them from the table, or
modify them so they have status of 'Terminated':

delete from fnd_concurrent_requests where status_code='T';

OR

update fnd_concurrent_requests
set status_code='X', phase_code='C'
where status_code='T';

+==+

References:

C:/…/1011248.6 10/10