Concurrent Manager FAQ's

Concurrent Manager Questions and Answers Relating to Generic Platform


PURPOSE
-------

To deliver a set of frequently asked questions against AOL FND Concurrent
Manager.


SCOPE & APPLICATION
-------------------

This reference would be most useful for one who is seeking additional
information regarding AOL FND Concurrent Manager issues.




FND Concurrent Manager FAQ's
-----------------------------

When would one be required to bounce (stop and restart) the Concurrent Manager?

When you modify the Printer Driver you have to restart the Manager which
runs the request which is attached to that Printer Driver, however,if you do
not know which manager then you have to restart the Internal manager because
the printer driver can be used by multiple managers and multiple requests.
If only a concurrent program definition is modified, running a verify on the
Internal Manager will pick up the changes without the need for bouncing the
manager.



Does the Internal manager schedule requests to be run or does it put requests
into queues to be run by other managers?

This is a very common misconception. The ICM really does not have any
such scheduling responsibilities. It has NOTHING to do with scheduling
requests, or deciding which manager will run a particular request.
Its function is only to run 'queue control' requests, which are
requests to startup or shutdown other managers. It is responsible for
startup and shutdown of the whole concurrent processing facility, and
it also monitors the other managers periodically, and restarts them if
they should go down. It can also take over the Conflict Resolution
manager's job, and resolve incompatibilities.


If the ICM itself should go down, requests will continue to run
normally, except for 'queue control' requests. You can restart it with
'startmgr', you do not need to kill the other managers first.



How can I check to see if a concurrent manager is running?

One way to see if a manager is running is to use the 'Administer
Concurrent Managers' form. Navigate to Concurrent->Managers->Administer.
You will see two columns labeled 'Actual' and 'Target'. The Target column
lists the number of processes that should be running for each manager
for this particular workshift. The Actual column lists the number of
processes that are actually running. If the Actual column is zero, there
are no processes running for this manager. If the Target column is zero,
then either a workshift has not been assigned to this manager, or the current
workshift does not specify any target processes. If the target column
is not zero, then the manager processes have either failed to start up,
or gone down. You should check the manager's logfile and the ICM
logfile. You can also search for OS processes using the 'ps' command.
It is possible for the form to be inaccurate, i.e. it may show actual
processes even though they are not really running. When in doubt, check for
processes at the OS level. On NT, you can check to see if the Concurrent Manager
service is running using the Services control panel.



Where do concurrent request or manager logfiles and output files go?

The concurrent manager first looks for the environment variable
$APPLCSF. If this is set, it creates a path using two other
environment variables: $APPLLOG and $APPLOUT
It places log files in $APPLCSF/$APPLLOG, output files go in
$APPLCSF/$APPLOUT

So for example, if you have this environment set:
$APPLCSF = /u01/appl/common
$APPLLOG = log
$APPLOUT = out

The concurrent manager will place log files in /u01/appl/common/log,
and output files in /u01/appl/common/out
Note that $APPLCSF must be a full, absolute path, and the other two
are directory names.

If $APPLCSF is not set, it places the files under the product top of
the application associated with the request. For example, a PO report
would go under $PO_TOP/$APPLLOG and $PO_TOP/$APPLOUT
Logfiles go to: /u01/appl/po/9.0/log
Output files to: /u01/appl/po/9.0/out
All these directories must exist and have the correct permissions.

Note that all concurrent requests produce a log file, but not necessarily
an output file.
Concurrent manager logfiles follow the same convention, and will be
found in the $APPLLOG directory



What are the logfile and output file naming conventions?

Request logfiles: l.req


Output files: If $APPCPNAM is not set: .
If $APPCPNAM = REQID: o.out
If $APPCPNAM = USER: .out


Where: = The request id of the concurrent request
And: = The id of the user that submitted the request


Manager logfiles:


ICM logfile: Default is std.mgr, can be changed with the mgrname
startup parameter
Concurrent manager log: w.mgr
Transaction manager log: t.mgr
Conflict Resolution manager log: c.mgr


Where: is the concurrent process id of the manager



Can I delete a concurrent manager?

You can disable the manager by checking the 'Enabled' checkbox, or
you can simply Terminate the manager and it will not run again unless
you reactivate it.
If it is really necessary, you can query the manager in the
'Define Manager' form, and delete the row. (It is recommended that you
DO NOT do this)

What is the function of the 'Conflict Resolution Manager'?

Concurrent managers read requests to start concurrent programs running. The
Conflict Resolution Manager checks concurrent program definitions for
incompatibility rules.

If a program is identified as Run Alone, then the Conflict Resolution Manager
prevents the concurrent managers from starting other programs in the same
conflict domain.

When a program lists other programs as being incompatible with it, the
Conflict Resolution Manager prevents the program from starting until any
incompatible programs in the same domain have completed running.


What is the 'Internal Scheduler/Prereleaser' manager?

The short name for this manager is FNDSCH. It is also known as the
Advanced Scheduler/Prereleaser Manager. This manager is intended
to implement Advanced Schedules. Its job is to determine when a
scheduled request is ready to run. Advanced Schedules were not fully
implemented in Release 11.0, they are implemented in Release 11.5,
but are not widely used by the various Apps products. General Ledger
uses FNDSCH for financial schedules based on different calendars and
period types. It is then possible to schedule AutoAllocation sets,
Recurring Journals, MassAllocations, Budget Formulas, and MassBudgets
to run according to the General Ledger schedules that have been
defined.

If financial schedules in GL are not being used then it is not a
problem to deactivate this manager.



What is the 'Internal Monitor' manager/service?

This manager/service is used to implement Distributed Concurrent Processing.
It monitors whether the ICM is still running, and if the ICM crashes,
it will restart it on another node.
You do not need to run this manager/service unless you are using Distributed
Concurrent Processing.


See the Installation manual and Sysadmin Guide for more info on DCP.



How do I check/set the PMON method?

To check the PMON method:
1) cd $FND_TOP/sql
2) sqlplus apps/apps @afimchk.sql
This will tell whether the internal manager is running, what the PMON
method is, and where the log file is.


To set the PMON method:
1) first shut the concurrent managers down
2) cd $FND_TOP/sql
3) sqlplus apps/apps @afimpmon.sql LOCK (or RDBMS)



How do I enable/disable the Conflict Resolution Manager?

Use the system profile option 'Concurrent: Use ICM'.
Setting this to 'No' (which is the default) allows the CRM to be started.
Setting it to 'Yes' causes the CRM to be shutdown and the Internal
Manager (ICM) will take over the conflict resolution duties. If the CRM will
not start (it is started automatically by the ICM), check this profile option.


Note that using the ICM to resolve conflicts is not recommended.
The CRM's sole purpose is to resolve conflicts, while the ICM has
other functions to perform as well. Only set this option to 'YES'
if you have a good reason to do so.



How do I clean out the Concurrent Manager tables?

Cleaning out the tables is a useful method of making sure that there
are no invalid statuses that can prevent the managers from starting.
Previously, this has been done by truncating fnd_concurrent_processes
and/or fnd_concurrent_requests. Truncation of the tables is a little
drastic, and can cause problems later when trying to purge requests,
not to mention losing all of the request information.


Run the script, cmclean.sql, article Note 134007.1 CMCLEAN.SQL - Non
Destructive Script to Clean Concurrent Manager Tables
It will make sure the relevant status codes are valid without
deleting any information.



How do I tell concurrent manager processes apart at the OS level?

Use: pf -ef | grep FNDLIBR


This will produce output like:


vd11 13703 13660 0 May 11 ? 0:01 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS/94A491A1000000000000000000
n1070161 24936 24927 0 Apr 29 ? 0:05 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000
n1070161 24938 24927 0 Apr 29 ? 0:06 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000
n1070161 24927 24922 0 Apr 29 ? 2:03 FNDLIBR FND CPMGR FNDCPMBR sysmgr
="" sleep=60 pmon=20 diag=N logfile=/u16/app


The last process, #24927, shows 'FNDLIBR FND CPMGR', this one is the
Internal Manager (ICM). Notice that it gives some of the parameters it
was started with, the other processes showing 'Concurrent_Processor'
are Standard manager processes. Notice that the ICM process is the
parent process of the Standard managers. (processes 24936 and 24938)

Other managers will have the name of the executable, like ARLIBR or
INVLIBR:

$ ps -ef | grep ARLIBR
vd11 13683 13660 0 May 11 ? 0:20 ARLIBR APPS/82A2A4940000000000000
000000000000000000000000000000000000000 AR ART


The Conflict Resolution manager will look like:

$ ps -ef | grep FNDCRM
n1070161 24941 24927 0 Apr 29 ? 1:17 FNDCRM APPS_APPDEMO/84BFBEB900000
0000000000000000000000000000000000000000000000



What is the syntax for controlling the concurrent manager using startmgr and
concsub in NT?

On NT, the concurrent manager is run as an NT service, you start and
stop the managers using the Services control panel.
See the Applications Installation manual for NT, Appendix A for
details. See pg. 5-9 of this manual for instructions on creating the
concurrent manager service.




Why am I seeing pinging entries like this in the ICM logfile?

PING (0.0.0.0): 56 data bytes
64 bytes from 192.75.91.2: icmp_seq=0 ttl=255 time=0.705 ms
64 bytes from 192.75.91.2: icmp_seq=1 ttl=255 time=1.120 ms
Process monitor session ended : 29-FEB-2000 10:38:43
64 bytes from 192.75.91.2: icmp_seq=2 ttl=255 time=0.985 ms
64 bytes from 192.75.91.2: icmp_seq=3 ttl=255 time=1.006 ms


Pinging other machines is used in Distributed Concurrent Processing.
This means you have DCP turned on, using the environment variable
APPLDCP. Set APPLDCP to OFF and restart the managers.



I hit the Restart button to start the Standard manager, but it still did not
start?

Telling a manager to restart just sets the status to Restart. The ICM
will start it the next process monitor session or the next time the
ICM starts. Use Activate to start a manager immediately.
When a manager is deactivated manually, the ICM will not restart
it, you will need to set it to Restart, or activate it manually.



How many rows are in FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESSES
tables?

When tables reach above 3000-4000 rows, the performance begins to
diminish. You may want to run Purge Concurrent Request on a regular basis,
dependant on the amount of requests being run.

The Purge Concurrent Requests job can be used to purge:
Requests, Mgr logs, and All requests depending on what is chosen.

Use the following options: Enter = All, Mode = AGE, Mode Value = 15

The std.mgr log continuously grows where it may good to
archive it regularly.



Any processes pending in Internal or Conflict Resolution Manager?

Best course of action before starting the Concurrent Managers is to cancel
any "Deactivate" or "Verify" jobs pending in the Internal Manager and place
any other pending jobs on hold.



How do I turn on transaction manager diagnostics?

Set the profile option 'Concurrent:Debug Flags' to 'TCTM1' at the site
level. This will cause transactions to make debug entries in the
FND_CONCURRENT_DEBUG_INFO table. Truncate this table before running a
tranasction, then select the entries from the table.
Starting the managers with diag=Y will also produce more information
in the transaction manager logfile.



How do transaction managers work?

Briefly:
(See the server documentation for details on the DBMS_PIPE package)
1) A tranasction manager is started on the concurrent processing
server, and periodically reads the pipe for incoming transactions.
2) A client program (usually a form) calls the
FND_TRANSACTION.SYNCHRONOUS function.
3) This function writes a message into the pipe containing the program
to be run and its parameters.
4) FND_TRANSACTION.SYNCHRONOUS begins reading a return pipe for the
return status.
5) The manager sees the message in the pipe, retrieves the program id
and parameters.
6) The manager runs the program with the specified parameters. The
program will be of type 'Immediate', so there will not be a
separate concurrent request run.
7) The program completes, and the manager packs its return status into
the return pipe.
8) FND_TRANSACTION.SYNCHRONOUS reads the return value and passes it
back to its caller.


Note that these events take place essentially simultaneously on the
client and server. This is a synchronous transaction because the
client waits for the server to return, or times out waiting for it.



Problem....

When you try to submit a request like Active users or
Active responsibilities, request gets submitted.

When we view the help requests, you find that it is
inactive / nomanager.

Within 12 to 15 seconds, you refresh-it gets completed.

Initially, you could find only inactive and we look at
the diagnostic- the concurrent manager assigned is not
picking up.

There is no specialization rules in any managers except
the include program this source.

Solution....

Most often when this occurs where a request goes
"inactive/no manager" and is then processed a short time
later, the solution is to either increase the cache size
for your Standard manger, or increase the actual number of
Standard manager processes.

Cache Size is set on the CONCURRENT/MANAGER/DEFINE form. Basically,
this regulates how many requests a manager will pick up for each
sleep cycle.


How do I process more concurrent requests concurrently?

The Concurrent Manager parameters, (Query the concurrent manager by
Login as Sysadmin, navigate -> Concurrent -> Manager -> Define and Query for
the relevant concurrent manager), should be modified to handle more
concurrent requests concurrently, this can be done in two steps:

(i) Increase the Number of Target processes for the manager

(ii) Change the cache size of the concurrent manager as this determines
how many requests will be evaluated by a manager at a time and should match the target (process) value as set above.



RELATED DOCUMENTS
-----------------
Note 1050938.6 What to Set $APPCPNAM for the Report Output File Naming Convention Format
Note 149600.1 FND_CONC_PP_ACTIONS and FND_RUN_REQ_PP_ACTIONS Growing Exponentially
Note 134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables