SAP does not start - Application Server Infrastructure - SCN Wiki

1 of 6

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

Getting Started Newsletters Store

Welcome, Guest

Login

Register

Search the Community

Application Server Infrastructure / Application Server Infrastructure / SAP Startup Framework

SAP does not start

Added by Koju Iijima, last edited by Koju Iijima on Sep 17, 2014
SAP system does not start.
We have received many incident reports (aka message) whose titles are SAP system does not start.
For SAP support engineers, this title has broad meanings. Since SAP system is based on the client server technology and consists of many processes or threads, there are many places to be checked
in order to understand what the exact problem you are facing.
What do you mean by SAP system does not start?
By reading the description of the incident report further, the reporters describe the situation like: .. the users cannot logon after restart.., .. the SUM tool says startup failure.., .. my message server
didnt come up.., ..I cannot see OS processes.., ..I started my own startup scripts but nothing happened.., and so forth.
Some of those descriptions are still quite vague, and some are quite to the point of the root cause.
First of all, please check this question:
Are Message Server, a dispatcher and work processes up and running?
Yes: -> this means that the startup of SAP system was finished successfully. It can be said that the problem is the logon itself, and not SAP system startup. If this is the case, then please avoid using the
title: SAP system does not start.
No: -> literally it means that SAP system DID NOT startup. Its startup sequence had some kind of problems.
Lets have a look this startup problem further.
This wiki aims to offer troubleshooting guides for startup problems. With this guide, we hope you can analyze and solve the problem by yourself. In some cases, you may still need to raise an incident ticket
(aka message). Then, we hope you can narrow down the problem and provide us detailed information so that we can instantly assist you.
Startup sequence
897933 - Start and stop sequence for SAP systems
In short, SAP system instances can be started up by the following sequence:
1. Database instance
2. Central instance or (A)SCS
3. Application servers (Dialog Instances)
*Database startup problem is not discussed in this wiki.
The following link explains standard methods to start SAP systems:
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/47/fd7230eca159e8e10000000a421937/frameset.htm
SAP MC, SAPMMC and startsap script needs SAP start service:
http://help.sap.com/saphelp_nw70ehp2/helpdata/en/b3/903925c34a45e28a2861b59c3c5623/frameset.htm
As you can see from above link, many processes are involved in the startup. Among those processes, the sapstartsrv process plays an important role of controlling a SAP instance.
The types of the processes which need to be executed in the system are declared in the instance profile (or start profile). Then, the sapstartsrv process starts up the specified processes.
During this startup sequence, each process generates its trace file on DIR_HOME directory.
Common problems and solutions
Quite common problems are related to the sapstartsrv process, especially following two symptoms: The process doesnt exist and The process exists but does not respond.
The process doesnt exist
On UNIX/LINUX, use an OS command ps to check the process:
ps ef | grep sapstartsrv
When the process exists, the above command shows like this:
abcadm 7012358

1 0 17:39:33

0:05 /usr/sap/ABC/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/ABC/SYS/profile/START_DVEBMGS00_walldorf -D -u abcadm

On WINDOS, use WINDOWS task manager tool to see if the service is running or not.

If the sapstartsrv process doesnt exist, please consult following SAP notes and start up.
1762827 - Startup of Instance Service failed in UNIX or Linux environment

The sapstartsrv process exists but does not respond

05/12/15 6:01 PM

SAP does not start - Application Server Infrastructure - SCN Wiki

2 of 6

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

The OS command PS or the taskmanager shows the process. But the sapstartsrv doesnt respond to any actions. Then, most likey, the UNIX domain socket (/tmp/.sapstream<port-no>) or Windows
named pipe (\\<host>\pipe\sapcontro_<xx>) may be broken. If it still exists, please manually delete. And start the sapstartsrv service again.
Other cases
Case 1:
Found the following error information in the startsap.log.
exec(): 0509-036 Cannot load program disp+work because of the following
errors:
rtld: 0712-001 Symbol ixml_iXMLParser_IsNormalizingAtt was referenced
from module disp+work(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol ixml_iXMLParser_SetNormalizingAtt was referenced
from module disp+work(), but a runtime definition
Root cause:
The library path was not configured correctly after upgrade.
Solution:
Configure the library path.
SAP Note 1265456 SAP system does not start after kernel upgrade
SAP Note 1104735 Upgrade to the new instance-specific directory on UNIX
Case 2:
The sapstartsrv service is up but the dispatcher doesnt startup.
sapcontrol -nr 00 -function GetProcessList
30.11.2013 18:46:21
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
disp+work, Dispatcher, GRAY, Stopped, , , 26680
igswd_mt, IGS Watchdog, GREEN, Running, 2013 11 30 12:57:32, 5:48:49,
26681
Root cause:
The zombie process of the dispatcher existed.
Solution:
Kill the zombie process.
Case 3:
On NT, frequently failover in few minutes.
Root cause:
The version of the sapstartsrv.exe was different between DIR_CT_RUN and DIR_EXECUTABLE.
Solution:
Copy the sapstartsrv.exe from DIR_CT_RUN to DIR_EXECUTABLE.
SAP Note 1043592 MSCS: Cluster Resource Monitor Crashes on W2K3 SP2
SAP Note 1345206 Handling and preventing sapstartsrv.exe corruptions
SAP Note 1375494 SAP system does not start after applying SAP kernel patch
Case 4:
Startsap.log says:
(1159276) Starting: local /usr/sap/ABC/SYS/exe/run/igswd_mt mode=profile pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf
(442404) Waiting for Child Processes to terminate.
(442404) **** 2010/04/10 08:07:11 Child 1155152 terminated with Status
255 . ****
(1155152) **** 2010/04/10 08:07:11 No RestartProgram command for
program 2 ****
The information above is very generic. stderr5 file shows more detailed information.
Command: /usr/sap/ABC/SYS/exe/run/igswd_mt
-mode=profile
pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf

05/12/15 6:01 PM

SAP does not start - Application Server Infrastructure - SCN Wiki

3 of 6

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

(1159276) **** Failed to Start Local Command. No such file or directory

Root cause:
The necessary files weren't copied correctly.
Solution:
Copy the igswd_mt to /usr/sap/ABC/SYS/exe/run
Case 5:
After kernel patching, the system doesn't come up but the startsap commands runs successful. The dispatcher was immediately down and no
work process is active.
Stderr2 says:
(1114238) New Child Process created.
(1114238) Starting local Command:
Command: dw.sapABC_DVEBMGS01
pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_walldorf

Fri Mar 5 08:59:56 2010

Profile configuration error detected, use temporary corrected setup
Shared Pool 40: ipc/shm_psize_40 = 2180000000 (too small)
Shared Pool 40: (smaller than min requirement 2186246040)
Shared Pool 40: (estimated size assumed 2190000000)
*** ERROR => Illegal ShmAdm Slot. Key=2104967296.
*** ERROR => ShmGet: Inv PoolKey=2104967296, Key=40, Flag=1, Ptr=1152921504606843304.
***
Root cause:
Shared memory area may need to be adjusted for new environment (i.e, after patching).
Solution:
Run sappfpar check pf=<profile> and adjust ipc/shm_psize_xx to suggested values.
Case 6:
The dispatcher went down right after startup. The dev_disp shows:
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG CPS=> DpLoopInit, ICU ( 3.4 3.4 4.1) [dpxxdisp.c 1706]
***LOG Q0K=> DpMsAttach, mscon ( CLMAGMACI) [dpxxdisp.c 12674]
*** DP_FATAL_ERROR => DpMsAttach: local hostname 'SRMAGMA98' is resolved
to loopback address (cf. SAP note 1054467 for details)

*** DISPATCHER EMERGENCY SHUTDOWN ***

Root Cause
Incorrect setup of the loopback address.

Solution:
#1054467 Local host name refers to loopback address

Case 7:
The dispatcher went down right after startup. The dev_disp shows:
*** ERROR => gateway (pid 29622344) died [dpxxdisp.c 16763]
*** DP_FATAL_ERROR => Gateway died with status 2 - I better exit now

*** DISPATCHER EMERGENCY SHUTDOWN ***

The gateway trace dev_rd shows:

***LOG Q0I=> NiIBindSocket: bind (67: Address already in use)
[nixxi.cpp 3740]
*** ERROR => NiIBindSocket: SiBind failed for hdl 9/sock 12
(SI_EPORT_INUSE/67; I4; ST; 0.0.0.0:3300) [nixxi.cpp

3740]

05/12/15 6:01 PM

SAP does not start - Application Server Infrastructure - SCN Wiki

4 of 6

***LOG S0V=> GwStopGateway, gateway stopped () [gwxxrd.c

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

13096]

Root Cause:
The service port 3300 was declared twice in the /etc/services file.

sapgw00

3300/tcp

sapgw00s

3300/tcp

Solutoin:
Make sure only one service is available.
sapgw00
#sapgw00s

3300/tcp
3300/tcp

Case 8:
The dispatcher went down right after startup. The dev_disp shows:
sapstart.log
(11925) Starting local Command:
Command: dw.sapPIP_DVEBMGS00
pf=/usr/sap/PIP/SYS/profile/PIP_DVEBMGS00_unjspfsappip

(11856) Waiting for Child Processes to terminate.

(11856) **** 2013/09/19 10:31:13 Child 11925 terminated with Status 0 . ****

dev_disp
*** ERROR => NiDgHdlBindName: invalid hostname 'localhost' (rc=-2;hdl 1) [nixx.c

4341]

*** ERROR => DpCommInit: NiDgHdlBindName failed: -8 [dpxxdisp.c 10339]

*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

Root Cause:
hosts file was modified and localhost was misconfigured.

Solution:
Configure localhost in the hosts file correctly.
Case 9:
The sapstartsrv is up.
When you execute sapcontrol without host opetion, you get: NIECON_REFUSED (WSAECONNREFUSED) error. However, it success with -host opetion.
For example:
sapcontrol -nr 00 -function GetVersionInfo
FAIL: NIECONN_REFUSED (WSAECONNREFUSED: Connection refused),
NiRawConnect failed in plugin_fopen()
but
sapcontrol -nr 00 -host saphost.sap.co:50013 -function GetVersionInfo -> works!
Root Cause:
Please refer SAP Note 2062508 sapcontrol commands fail with error "NIECONN_REFUSED (WSAECONNREFUSED: Connection refused)" during upgrade
Solution:
Comment out parameter service/hostname for the SAP start service and restart sapstartsrv.

The sapstartsrv exisits. -function GetProcessList doesn't show programs like dispatcher, gateway, etc.
In normal case, you will see something like this:
---sapcontrol -nr 00 -function GetProcessList

05/12/15 6:01 PM

SAP does not start - Application Server Infrastructure - SCN Wiki

5 of 6

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7326
disp+work, Dispatcher, GREEN, Running, Message Server connection ok, Dialog Queue time: 0.00 sec, 2014 02 25 05:45:24, 0:00:53, 7327
igswd_mt, IGS Watchdog, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7328
---->here, you can see programs (message server, disp+work and IGS) are up and running.
But in this problematic situation, something like....
---sapcontrol -nr 00 -function GetProcessList
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
-----> No programs are listed .
This is because 'Start_Program_n' command with the program is missing in the profile file which sapstartsrv starts with.
For this 'Start_Programm_n", please refer the following link:
http://help.sap.com/saphelp_nw74/helpdata/en/9c/ac0d50a40af26be10000000a423f68/frameset.htm
(NOTE!!! since kernel 74x release, 'Start Profile' is obsolated. You cannot use 'Start Profile' like ealier release. All the information in Start Profile can be marged to Instance Profile)

After upgrading to 74x system, SAP doesn't startup.

In most cases, the reason is the sapstartsrv is pointing to Start Profile. Start Profile is obsolated since 74x kernel. You need to recofigure your profile.
Delete Start Profile physically from 'profile' dir.
In UNIX, adjust /usr/sap/sapservices file.
In WINDOWS, adjust the SAP service.
Delete UNIX domain socket or Window's named pipe, and restart sapstartsrv.

UNIX Domain Socket .sapstream5xx13 was not enabled...

Case 1:
sapstartsrv process exists but the corresponding UNIX Domain Socket doesn't exist.
Checki sapstartsrv.log. It ends at CCMS agent initialization phase:
cat sapstartsrv.old
--------------------------------------------------trc file: "sapstartsrv.log", trc level: 0, release: "741"
--------------------------------------------------pid 2134
[Thr 01] Mon Sep 15 10:09:13 2014
HistoryLog_Init: logfile (/usr/sap/ABC/SCS00/work/history.glf) already exists, check parameter
No halib defined => HA support disabled
CCMS agent initialization for instance type SCS: return code 0.
-->that's all.... sapstartsrv should initialize sapcontrol webservice....
Normally, you should see something like:
CCMS agent initialization for instance type SCS: return code 0.
CCMS agent disabled by profile parameter ccms/enable_agent = -1.
Initializing SAPControl Webservice
Auto PSE update thread started
Starting AutoRestart thread
Starting WebService SSL thread
AutoRestart thread started
Starting WebService thread
Webservice thread started, listening on port 55013
Webservice SSL thread started, listening on port 55014
Webservice SSL thread using default SAP SSL credential
Trusted https connect via Unix domain socket '/tmp/.sapstream55014' enabled.
Trusted http connect via Unix domain socket '/tmp/.sapstream55013' enabled.
Root Cause:
CCMS agent has been running already by saphostcontrol.
Solution:
set ccms/enable_agent = 0 in the instance profile.
Case 2:

05/12/15 6:01 PM

SAP does not start - Application Server Infrastructure - SCN Wiki

6 of 6

http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

A trace file called sapstartsrv_ccms.log was generated in work dir. The content of i ends with the information:
INFO: Checking lock file.....
Root Cause:
Please refer SAP note 1916333 sapcontrol error "FAIL: NIECONN_REFUSED" due to agent.log file
Solution:
set ccms/enable_agent = 0 in the instance profile.

No labels

Contact Us
Privacy

SAP Help Portal

Terms of Use

Legal Disclosure

Copyright

Follow SCN

05/12/15 6:01 PM