1.Check /etc/hosts file to verify it has correct syntax: <IP address> <hostname.

domain> <hostname> and for one hostname don't have multiple entries. 2.Before starting the concurrent managers using adcmctl.sh,Make sure all concurrent managers process FNDLIBR have been shutdown at the OS level. for example: >ps -ef|grep $USER|grep FNDL if there still have FDNLIBR processes still running. >kill -9 PID PID form output of the command ps in previous step. 3.Changing status of Concurrent Requests from Pending to Inactive. At form [Administer Concurrent Managers(FNDCPVCM)],select the concurrent manager which pending rrquests is not zero and click button [Requests] in the lower right corner. in the pop up windows,recording the request IDs,and then using specific Request in Find Request window search these request,(notice the field [Select the number of Days to view] should be set a properly value.),in Requests window,click [Hold Request] to change the phase of these request to [Inactive] and status to [On Hold] . if ALL concurrent managers been shown in form [Administer Concurrent Managers(FNDCPVCM)] with zero or null pending requests,then restart CM,after the CM processes have been started normally, then in request windows,you can click [Move Hold] to change the phase of these request back to [pending]. 4.Check the value of system profile [Concurrent:GSM Enabled] at Site level. if the value of [Concurrent:GSM Enabled] is Y,changing it to N and restart concurrent managers,if concurrent managers can start normally,changing it back to Y and restart CM again. 5.Make sure customize env file has been added in env file which used in Concurrent Manager start/stop scripts adcmctl.sh. >cat $COMMON_TOP/admin/scripts/$CONTEXT_NAME/adcmctl.sh|grep ENVFILE= if your environment have user defined customize env files,please sure they have been added in env file used in adcmctl.sh. 6. Relink executable program under $FND_TOP/bin using adadmin. adadmin-->2. Maintain Applications Files menu-->1. Relink Applications programs when ask specific products,input:FND Enter list of products to link ('all' for all products) [all] : FND 7.Rebuild the concurrent manager views. As applmgr run the following from the OS:

This is non-destructive. Concurrent Manager views can be rebuilt by running the following command at the command line: Ensure that concurrent managers are shutdown. FNDLIBR FND FNDCPBWV apps/apps SYSADMIN 'System Administrator' SYSADMIN Restart the concurrent managers. 8.The Profile Option 'Concurrent: OPS Request Partitioning' may be set incorrectly. This profile option should always be set to OFF, regardless of whether you are running OPS(RAC) or not, because the profile is obsolete. 9. The System Profile Option: Concurrent Active Requests is possibly to 0. a. Log into Oracle Applications as SYSADMIN. b. Select System Administrator responsibility. c. Navigate to PROFILE > SYSTEM. d. Query for %CONC%ACTIVE%. e. Change the profile option for 'Concurrent: Active Request Limit' to Null (blank). f. Exit Oracle Applications and log in again for the change to take affect. g. Run a new concurrent request. 10. The Concurrent managers were brought down, while an outstanding request was still running in the background. In which case, update the FND_CONCURRENT_REQUESTS table as follows: sql> update fnd_concurrent_requests set status_code='X', phase_code='C' where status_code='T'; sql> commit; STATUS_CODE SQL> select lookup_code,meaning 2 from fnd_lookups 3 where lookup_type = 'CP_STATUS_CODE'

4 order by lookup_code; LOOKUP_CODE MEANING

--------------- -------------------A B C D E G H I M P Q R S T U W X Z Waiting Resuming Normal Cancelled Error Warning On Hold Normal No Manager Scheduled Standby Normal Suspended Terminating Disabled Paused Terminated Waiting

PHASE_CODE SQL> select lookup_code,meaning 2 from fnd_lookups 3 where lookup_type = 'CP_PHASE_CODE'; LOOKUP_CODE -------------C MEANING

-------------------Completed

I P R

Inactive Pending Running

11. The control_code for concurrent_queue_name = 'FNDCRM' is 'N' in the FND_CONCURRENT_QUEUES table, which means 'Target node/queue unavailable'. This value should be NULL (CRM is running; target and actual process amount are the same), or 'A' ('Activate concurrent manager' control status). Set the control_code to 'A' in fnd_concurrent_queues for the Conflict Resolution Manager: a. Logon to Oracle Applications database server as 'applmgr'. b. Verify the Applications environment is setup correctly ($ORACLE_HOME and $ORACLE_SID). c. Logon to SQL*Plus as 'APPS' and run the following SQL statement: update fnd_concurrent_queues set control_code = 'A' where concurrent_queue_name = 'FNDCRM'; commit; d. Verify the status of the concurrent managers through the Concurrent -> Manager -> Administer form. If the CRM is still not active, bounce (deactivate, activate) the Internal Concurrent Manager. This is done through the Concurrent > Manager > Administer form from the 'System Administrator' responsibility. It can also be done through the CONCSUB command at the command level. Setting the control_code to 'A' in the fnd_concurrent_queues table for the Conflict Resolution Manager indicates that this concurrent manager is to be activated with the parameter values specified through this table for this manager (MAX_PROCESSES, CACHE_SIZE, etc). Concurrent Manager [Conflict Resolution Manager] and [Scheduler/Prereleaser Manager] cannot be Restart or Active in the form [Administer Concurrent Managers(FNDCPVCM)], so we need update control_code to do this job. queue name of Conflict Resolution Manager is FNDCRM. queue name of Scheduler/Prereleaser Manager is FNDSCH. Control_code SQL> select lookup_code,meaning 2 from fnd_lookups

3 where lookup_type = 'CP_CONTROL_CODE' 4 order by lookup_code; LOOKUP_CODE MEANING

--------------- -------------------A B D E N O P Q R T U V X Activating Activated Deactivating Deactivated Target node/queue unavailable Suspending concurrent manager Suspended Resuming concurrent manager Restarting Terminating Updating environment information Verifying Terminated

12. What is the cache size? Try increasing the cache size then stop/restart the concurrent managers. If concurrent requests are rarely prioritized and there are managers that service short-running requests, consider setting the cache size to equal at least twice the number of target processes. This increases the throughput of the concurrent managers by attempting to avoid any sleep time. For example: If more than one manager or worker processes the same type of requests with only a small cache size, it may be unable to process any jobs in a single processing cycle, because other processes have already run the cached requests. When this happens, it is important to note that the manager will sleep before refreshing its cache. To increase manager throughput where there are sufficient requests of the required type in the queue, increase the cache size to improve the chance of the manager finding work to process and thus avoid having to enter a sleep phase. TIP: Ensure that the system is not resource-constrained before attempting to increase the rate of concurrent processing in this way, otherwise,

these changes may actually reduce concurrent processing throughput because jobs take longer to run. Help of Cache size Cache Size (Concurrent Manager only) Enter the number of requests your manager remembers each time it reads which requests to run. For example, if a manager's workshift has 1 target process and a cache value of 3, it will read three requests and wait until these three requests have been run before reading new requests. In reading requests, the manager will only put requests it is allowed to run into its cache. For example, if you have defined your manager to run only Order Entry reports then the manager will put only Order Entry requests into its cache. If you enter 1, the concurrent manager must look at its requests list each time it is ready to process another request. By setting the cache size at a higher number, the concurrent manager does not have to read its requests list each time it runs a request. However, the manager does not recognize any priority changes you make for a particular request if it has already read that request into its cache. Further, even if you give a higher priority to a new request, that new request must wait until the buffer is empty and the manager returns to look at the requests list. That request may have to wait a long time if you set the buffer size to a high number. You should use cache size to tune your concurrent managers to work most efficiently for you site's needs. If your organization tends to reprioritize jobs going to a certain manager, that manager should have its buffer size set fairly low. Suggestion: Enter a value of 1 when defining a manager that runs long, time-consuming jobs, and a value of 3 or 4 for managers that run small, quick jobs. 13.Concurrent Manager [Output Post Processor] cannot be started with Actual is 0 and Service Info %%service threads running. Select [Output Post Processor] in Administer and click [Processes] button in lower right corner,verify [Start Date],if it early than restart date,you should Terminate it first in main windows and the restart it. 14.OAM Generic Collection Service:HOSTNAME is showing status:Deactivated in FNDCPVCM and "The target node/queue unavailable" in OAM. OAM access path:OAM-->Site Map -->Application Services -->Generic Services --> Select OAM Generic Collection Service -->View Details. using below sql to verify TARGET_NODE=<Right Node> and CONTROL_CODE=NULL,if not,update. SQL> select CONCURRENT_QUEUE_NAME, CONTROL_CODE, TARGET_NODE, NODE_NAME from FND_CONCURRENT_QUEUES where concurrent_queue_name like 'OAMGCS_%'; 15.if "The target node/queue unavailable" had been shown after cloning or Adding new node.

>sqlplus apps/appsuat SQL*Plus: Release 10.1.0.5.0 - Production on Mon Mar 23 15:48:04 2009 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, OLAP and Data Mining options SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN; PL/SQL procedure successfully completed SQL> update icx_parameters set session_cookie_domain=NULL; 1 row updated SQL> commit; Commit complete SQL> exit after execute FND_CONC_CLONE.SETUP_CLEAN,firstly run Autoconfig on ALL DB tier and then all APP tier. 16.from note:738405.1,if the Internal Concurrent Manager starts, but all others managers have a status of 'Target queue/node unavailable'. and the log file of internal Concurrent Manager shows the following errors: Internal Concurrent Manager found node to be down . you should check if ping command can be used in OS level. >ping hostname 17.Note:104541.1 Troubleshooting (Concurrent Manager Generic) 18.Note:105133.1 Concurrent Manager Questions and Answers Relating to Generic Platform 19.Note:134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables @END The new version of this doc can be find in:http://docs.google.com/Doc?id=dd9hpwxq_14dhrvqv3z