HP Data Protector collecting debug files.

Technical white paper Table of contents

Summary ...................................................................................................................................... 3 Change the default location to store debug files ................................................................................ 3 Windows host ........................................................................................................................... 3 Set the variable OmniTmp in the registry ......................................................................................... 3 Set the variable OB2DBGDIR in the omnirc file ................................................................................ 3 Redirect the files using a path prefix : ............................................................................................. 4 Linux and Unix host .................................................................................................................... 4 Methods to start creating debugs and where to define the debug options. ............................................ 4 From the GUI ............................................................................................................................. 4 Scheduled debugging................................................................................................................. 4 From the command line............................................................................................................... 5 Enabling the trace file ................................................................................................................. 5 For Windows ............................................................................................................................... 5 For Unix/Linux ............................................................................................................................. 6 The omnirc ................................................................................................................................ 6 The cell server omnirc file .............................................................................................................. 6 The client omnirc file ..................................................................................................................... 6 Debug options ............................................................................................................................... 6 Range....................................................................................................................................... 7 Circular debugs ......................................................................................................................... 7 Timestamp in seconds and milliseconds ........................................................................................ 8 Debug files in Unicode format ..................................................................................................... 8 Postfix ....................................................................................................................................... 8 Program and hostname ............................................................................................................... 8 Debug definition changes and re-activation. .................................................................................. 9 Special debugs .......................................................................................................................... 9 Inet debugs ............................................................................................................................... 9 On Windows host: ....................................................................................................................... 9 On UNIX host ............................................................................................................................ 10 On Linux host ............................................................................................................................. 10 CRS and MMD debugs ............................................................................................................. 10 On Unix/Linux CS ...................................................................................................................... 10 For cluster configurations ............................................................................................................. 11 Omnitrig debugs ...................................................................................................................... 11 On Windows ............................................................................................................................. 11 On Unix/Linux ........................................................................................................................... 11 What debug files are needed? ...................................................................................................... 11 General debugs ....................................................................................................................... 11 Veagent debug log files ............................................................................................................ 11 VMware VDDK log files ............................................................................................................ 12 VMware log files...................................................................................................................... 12 VMware ESX(i) log files ............................................................................................................ 13 vCenter Server trivia log files ..................................................................................................... 14 How to collect debugs for MC/Serviceguard integration .............................................................. 14 Collecting CRS only debugs......................................................................................................... 14 Collecting CRS and MMD debugs ................................................................................................ 15 Collecting all debugs from cluster nodes excluding CLI and GUI debugs ........................................... 15 Disabling debugs ....................................................................................................................... 15 Collecting omniinet debugs.......................................................................................................... 15 Media Agent debugs ................................................................................................................ 15 Information required for Cell Manager and RDS problems. ........................................................... 17 The following information should be obtained ................................................................................ 17
Version: 1.03

See Appendix A for a list of helpful data for RDS problems where DP runs on HP-UX.......................... 17 Appendix A Data list for RDS problems on HP-UX........................................................................... 17 Appendix B Data list for RDS problems on Windows ...................................................................... 18 Information required for Disk and Media Agent Issues .................................................................. 19 Problem Description .................................................................................................................... 19 Collect Debugs........................................................................................................................... 20 Collect Matching Session Report. ................................................................................................. 20 Collect Supporting Data. ............................................................................................................. 20 Run the vbda as standalone command .......................................................................................... 20

Summary
Each Data Protector program can produce debug files. They contain messages that can be used in the reconstruction of the program execution. The debug trace files are meant to inform the support engineering, how and why something happened?

Where can I store debug files Methods to start creating debug files What kind of options are available when creating debug files What specific debug files need to be created for further troubleshooting

Change the default location to store debug files

Sometimes the space in the default repository is insufficient and debug files need to be redirected to another repository.

Windows host
By default the debug files are stored under the Omniback tmp directory. C:\ProgramData\OmniBack\tmp (for Windows vista, 2008, 7) C:\Program Files\OmniBack\tmp (older Windows versions) Set the variable OmniTmp in the registry One important note: the destination directory must exist and the permissions to the full path need to be correct for the process writing the debugs. Commands to query, add or delete the registry entry: reg add HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp" /d "C:\temp\debug" reg query HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp" HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\OpenView\OmniBackII\Common OmniTmp REG_SZ C:\temp\debug reg delete HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp" Set the variable OB2DBGDIR in the omnirc file
for Windows 2008 and vista omnirc is now in C:\ProgramData\OmniBack for older Windows versions (2003) omnirc is C:\Program Files\OmniBack

Set in the omnirc file OB2DBGDIR=C:\temp\debug

Redirect the files using a path prefix : omnib -datalist test -debug 1-500 c:\temp\output.txt

Linux and Unix host

For Linux, HP-UX debug files are stored by default in the root directory /tmp Destination is specified in the debug option, as prefix to the debug file name: -debug "1-99 104-140" /xyz/DBG.txt Or with a .omnirc variable OB2DBGDIR=/mount/depot

Methods to start creating debugs and where to define the debug options.
From the GUI

Note that only range and postfix can be defined. Those generic debug files may be created on several host.

Scheduled debugging
Specific backup specifications can run in debug mode when scheduled. To enable debugs open the corresponding file in the server config schedule folder. For barlist open Barschedules, select the type of barlist. e.g. on W2003: C:\Program Files\OmniBack\Config Server\Barschedules\MSSQL

on Windows 2008 and later: C:\ProgramData\OmniBack\Config\Server\Barschedules\VEAgent on linux/unix: /etc/opt/omni/server/barschedules/oracle8 Look for the barlist specification, copy it first and edit the file. Add debug 1-500 full.txt as first line. -debug 1-500 full.txt -incr -starting 8 5 2012 -every -day -at 14:30 For other backups go to the Schedules folder, and copy and edit the corresponding backup specification file. Add debug 1-500 full.txt as first line. To stop the debugging, remove the entry or restore the backup.

From the command line

Add option debug [range] postfix omnib -debug 1-99 debug.txt -filesystem miki:/ -tree ...

Enabling the trace file

The trace file is located under <Data_Protector_home>\Config\Server\Options Typical configurations are available in the trace file. Read also the header for more details. For Windows
#--------------------------------------------------------------------------# Filename: <Data_Protector_home>\Config\Server\Options\trace # # $Revision: 12151 $ # # This file is a template for Data Protector debugging. It contains many # different debugging options for collecting debug files for different # Data Protector modules. # # How to use it? # # 1. Uncomment line(s) containing the debug options (make sure only one # of each options is uncommented!): # # ranges= # postfix= # select= # # 2. Save changed trace file (this file) # # 3. Make sure no Data Protector session is running # # 4. Restart the crs daemon to apply debug options: # # Settings / Control Panel / Services / Data Protector CRS [Stop] # Settings / Control Panel / Services / Data Protector CRS [Start] # # 5. To turn the debugging off, set all debugging options to empty string: # # ranges=

# postfix= # select= # # and restart the crs daemon the same way as in 4. #---------------------------------------------------------------------------

For Unix/Linux
#--------------------------------------------------------------------------# Filename: /etc/opt/omni/server/options/trace # # $Revision: 12151 $ # # This file is a template for Data Protector debugging. It contains many # different debugging options for collecting debug files for different # Data Protector modules. # # How to use it? # # 1. Uncomment line(s) containing the debug options (make sure only one # of each options is uncommented!): # # ranges= # postfix= # select= # # 2. Save changed trace file (this file) # # 3. Make sure no Data Protector session is running # # 4. Restart the crs daemon to apply debug options: # # /opt/omni/lbin/crs -redebug # # 5. To turn the debugging off, set all debugging options to empty string: # # ranges= # postfix= # select= # # and restart the crs daemon the same way as in 4. # #---------------------------------------------------------------------------

The omnirc
The cell server omnirc file Run debugs on the cell server or on a specific client OB2DBG=1-500 MA.txt 'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com' The client omnirc file When a program starts it will always verify is a omnirc variable is set locally. Run debugs only locally on the client OB2DBG=1-500 bma.txt 'BMA,UMA' Note: always restart the dataprotector inet service omniInet when removing the OB2DBG variable.

Debug options
The options are always a combination of

Range and options Postfix Module and host selection Re-Activate the change

Almost all Data Protector commands can be started with an additional -debug parameter that has the following syntax: -debug 1-500[,C:n][,T:s][,U] XYZ [host] where:

Range
1-500 is the debug range. Specify an extended range when instructed. Specify optional parameters as a part of the range parameter, separated by commas.

Examples: When setting a large range the debug files will be large. Make sure there is enough space in the debug files repository. The range can be split. The separator can be a , or a space within a double quoted string. Valid ranges are: -debug "1-99 104-140" debug.txt -debug "1-99,104-140" debug.txt -debug 1-99,104-140 debug.txt Range 11, range 30 and 31, range 60 up to 63 and range 131. OB2DBG=11,30-32,60-64,131,C:1000000 scsi.txt 'BMA' One single range, 240 OB2DBG=240 cell_info.txt 'VEPA_BAR'

Circular debugs
C:n limits the size of debug files to n kilobytes. The minimum value is 4 (4kB) and the default value is 1024 (1 MB).

Examples: Select a range and make it circular 11,30-32,60-64,131,C:1000000 OB2DBG=1-99,C:2000000 MA.txt 'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com' ranges=1-500,C:100000 here the circular debug is for 100000 kBytes Note: One Gigabyte is 1024*1024=1048576 kBytes. In the trace file

ranges=1-500,C:2000000 postfix=BMA.txt select=BMA@system.domain Note: Only use circular debugs if you have limited space.

Timestamp in seconds and milliseconds

T:s is the timestamp resolution, accepted values are 0,1 and 1000, where the default value is 1, 1000 means the resolution is one millisecond and 0 means timestamps are turned off.

Examples: OB2DBG=11,30-32,60-64,131,C:1000000,T:1000 scsi.txt 'BMA' The timestamp will have a 3 digit postfix with milliseconds. 2012-04-09 10:39:03.054

Debug files in Unicode format

U is the Unicode flag. If it is specified, the debug files on Windows are written in the Unicode format.

Examples: OB2DBG=11,30-32,60-64,131,C:1000000,T:1000,U scsi.txt 'BMA'

Postfix
XYZ is the debug postfix, for example My_debug.txt.

Note: the postfix can be used to redirect the debug files to another directory. The destination directory must exist and the permissions to the full path need to be correct for the process writing the debugs.

Program and hostname

host is a list of clients where debugging is turned on. Use this option to run the debugging only on the clients specified. Delimit multiple clients by spaces. Enclose the list in quotes, for example: "computer1.company.com computer2.company.com". (single or double quotes can be used)

Examples: In the Cell Server omnirc, create BMA debug files on all MA gent OB2DBG=11,30-32,60-64,131 bma.txt 'BMA' In the Cell Server omnirc, create BMA and UMA debug files on dpsupport.hp.com only OB2DBG=1-99,C:2000000 MA.txt 'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com' In the client omnirc,e.g. create only VEPA debugs on the backup host

OB2DBG=1-199,240 VEPA.txt VEPA_BAR,VEPALIB_VMWARE_EXECUTION_THREAD,VEPALIB_VMWARE,VEPALIB_VMWARE_THR EAD

Debug definition changes and re-activation.

When changing the OB2DBG options, restart DP services on the Cell server or omniinet services on the client. Also when removing the entry in the omnirc file restart DP services on the Cell server or omniinet services on the client. The client omniinet service will inherit the settings from the cell server, changing a setting on a CS will not remove it from the client even if the setting was removed from the preference debug tab. unless the omniinet is restarted on the client. So it is always safe to clean up the CS and agent when trace activities are terminated.

Special debugs Inet debugs

On Windows host: Check for the omniInet service

Stop the service and enter start parameters debug 1-500 inet.txt Select start

Now the omniIinet service is restarted in debug mode.

Note this is a non-persistent setting and is removed with the next omniinet startup. On UNIX host Edit the /etc/inetd.conf file: Modify inet with the debug option omni stream tcp6 nowait root /opt/omni/lbin/inet inet log /var/opt/omni/log/inet.log -debug 1-500 inet.txt Save the file and run the /etc/inetd -c command to apply the changes. Note that this change is persistent. On Linux host Go to /etc/xinetd.d Edit omni server_args = inet -log /var/opt/omni/log/inet.log -debug 1-500 inet.txt Reload services /etc/init.d/xinetd reload Restart services /etc/init.d/xinetd restart Note that this change is persistent.

CRS and MMD debugs

Media Management Daemon (MMD) is started as a part of the CRS. To create debug files for Media Management, the CRS has to be started in debug mode. On Windows CS Launch the Windows Service Control Manager and restart the Data Protector CRS service with the following startup parameters:-debug 1-500 crs.txt

On Unix/Linux CS For HP-UX, first terminate the CRS using the command: /opt/omni/lbin/crs -shutdown

Restart the CRS with the debug option: /opt/omni/lbin/crs -debug "1-500" crs.txt Note the range is here an example. For cluster configurations Update the trace file for CRS debugging and restart the CRS service as explained in the header of the trace file. On Windows Server 2008 and later: Data_Protector_program_data\Config\server\options\Trace On other Windows systems: Data_Protector_home\Config\server\options\Trace HP-UX/Linux /etc/opt/omni/server/options/trace

Omnitrig debugs
On Windows Omnitrig is running as a part of CRS service. For testing and support purpose it can be started manually as any ordinary command from the command line: Omnitrig -debug 1-500 DBG On Unix/Linux Omnitrig is the following line into the crontab: 0,15,30,45 * * * * /opt/omni/sbin/omnitrig If the line is changed into something like this: 0,15,30,45 * * * * /opt/omni/sbin/omnitrig -debug 1-500 omnitrig.txt The following sequence of commands will do: $ crontab -l > FILE $ vi FILE $ crontab FILE

What debug files are needed?

General debugs
In most cases general debugs are 1-500 range. The full debugs can be large, specific with DA and MA debug files.

Veagent debug log files

If a problem is related to the VEAgent backup host the following general setting is recommended. From the GUI Preferences Debug tab -debug 1-199 If the problem is within the VEAgent, the unwanted BMA debugs will be very large. In order to limit the debugs the following is recommended. Start the inet debugs on the VEAgent backup host, add startup parameter -debug 1-500 inet.txt and restart omniInet.

11

Create a VEAgent backup host omnirc file or add the following line: OB2DBG=1-199,240 VM.txt "VEPA_BAR,VEPALIB_VMWARE_EXECUTION_THREAD,VEPALIB_VMWARE,VEPALIB_VMWARE_THR EAD" The range 1-199 is sufficient; the 240 range is adding the omni_cell content in the vepa_bar debug file. If networking details are needed use the following range 0-199,240-270

VMware VDDK log files

When the vmware integration fails the vddk log files could reveal more information on the root case. To enable, on the VEagent backup host, go to C:\ProgramData\OmniBack\Config\client Now edit the file vepa_vddk.config and change the LogLevel to the highest, 6. vixDiskLib.transport.LogLevel="6" The log file is dumped in the session report. vixDiskLib.transport.LogLevel = <logLevel> 0: Quiet 1: Panic 2: Error 3: Warning 4: Informational 5: Verbose 6: Trivia

VMware log files

The Management agent (hostd), VirtualCenter Agent Service (vpxa), and VirtualCenter (vpxd) logs are automatically rotated and maintained to manage their growth. Information in the logs can be lost if the logs are rotated too quickly. Note: Rotation means the turn over of files. For example, if you set the maximum number of log files to 10, after every 10 log files, the numbering restarts at 0. Each logs are found at these locations (note that the log files are physical on datastore): hostd o o vpxa o o

In ESX/ESXi 4.x hosts, hostd logs are located at /var/log/vmware/ In ESXi 5.0 hosts, hostd logs are located at /var/log/

In ESX/ESXi 4.x hosts, vpxa logs are located at /var/log/vmware/vpx/ In ESXi 5.0 hosts, vpxa logs are located at /var/log/

vpxd logs are located in %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs, which translates to: o C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs in Windows 2003

C:\ProgramData\VMware\VMware VirtualCenter\Logs in Windows 2008

This article provides steps to increase the size and number of the hostd, vpxa, and vpxd logs so that additional data is saved. This data may be useful for troubleshooting purposes.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004795

VMware ESX(i) log files

The esx(i) host has log files of all activities performed, e.g. snapshot creation, removal etc. The log files are text files starting with hostd and are zipped once full. hostd.log is the active log file. The files are located physical on datastore (e.g. /var/log ->/scratch/log->/vmfs/volumes/4e265cdb6b91f4b2-bc38-e4115b13545a/log) /vmfs/volumes # find . | grep hostd ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.log ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.5.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.4.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.3.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.2.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.1.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.0.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.9.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.8.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.7.gz ./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.6.gz Tip: Best is to copy/move the files into an browsable datastore, then its easy to download and to remove the files afterwards.
E.G. /vmfs/volumes/4e2090d4-3fd60a10-eb80-e4115bb0ff04/log # cp hostd.* /vmfs/volumes/esx10_int

2012-02-21T17:44:26.883Z [3A4F5B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=31603181-91] Create snapshot request received: _DP_VEPA_SNAP_, Created by Data Protector A.07.00 on 10/02/12 09:38:41, false, false 2012-02-21T17:44:26.883Z [3A4F5B90 info 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=31603181-91] State Transition (VM_STATE_ON -> VM_STATE_CREATE_SNAPSHOT) . 2012-02-21T04:50:28.004Z [39EB8B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Remove snapshot request received: _DP_VEPA_SNAP_, 0

13

2012-02-21T04:50:28.004Z [39EB8B90 info 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] State Transition (VM_STATE_ON -> VM_STATE_REMOVE_SNAPSHOT) 2012-02-21T04:50:28.013Z [FFA23AC0 info 'Libs' opID=b341f405-e5] SNAPSHOT: SnapshotDelete '/vmfs/volumes/4e2151a2-f70c1db0-3535e4115bb0ff00/g53/g53.vmx' : 9 2012-02-21T04:50:28.057Z [FFA23AC0 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Consolidate disks after snapshot removal.. 2012-02-21T04:50:28.057Z [FFA23AC0 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Consolidate disks job number 11534487... 2012-02-21T04:50:29.691Z [390B6B90 verbose 'Default'] Job cb 11534487 received 2012-02-21T04:50:29.691Z [39EB8B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx'] Done disk consolidation. ter.

vCenter Server trivia log files

Caution: Enabling trivia or verbose logging for a longer duration might cause performance degradation on vCenter Server. Only enable trivia or verbose logging for troubleshooting purposes. VMware highly recommends reverting to default login (info level) immediately after the troubleshooting is complete. VMware does not recommend trivia/verbose logging to be enabled in a production environment. Perform the steps in this article only after consulting VMware technical support. For more details:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001584

Enabling trivia level logging is normally done by clicking: Administration > vCenter Server Settings > Logging Options > Trivia for vCenter Server 4.x/5.x

How to collect debugs for MC/Serviceguard integration

Note that these instructions are applicable when Data-Protector cell manager is running as a virtual package. If the Data-Protector client is a virtual package, then it is no different from any other client. Collecting CRS only debugs On the node where DP package is running (use cmviewcl -v), edit /etc/opt/omni/server/options/trace Example: ranges=1-99 104-500 postfix=/tmpdir/DBG select=CRS Get CRS to reread the trace file # /opt/omni/lbin/crs -redebug Debugs will be generated under /tmpdir/*DBG

Collecting CRS and MMD debugs On the node where DP package is running (use cmviewcl -v), edit /etc/opt/omni/server/options/trace Example: ranges=1-99 104-500 postfix=/tmpdir/DBG select=CRS MMD Restart Data-Protector daemons # cmhaltpkg <package_hostname> # cmrunpkg [-n <node_name>] <package_hostname> Debugs will be generated under /tmpdir/*DBG Collecting all debugs from cluster nodes excluding CLI and GUI debugs On the node where DP package is running (use cmviewcl -v), edit /etc/opt/omni/server/options/trace Example: ranges=1-99 104-500 postfix=/tmpdir/DBG select=sgnode1.abc.com sgnode2.abc.com IF you don't need MMD and KMS debug, just run the redebug # /opt/omni/lbin/crs -redebug (does generate MMD or KMS debugs) If you need MMD and KMS debugs, restart Data-Protector daemons # cmhaltpkg <package_hostname> # cmrunpkg [-n <node_name>]<package_hostname> Debugs will be generated under /tmpdir/*DBG Disabling debugs Edit trace file and delete or comment out the statements included previously If you enable debugging using redebug option then do the same again. # /opt/omni/lbin/crs -redebug If you enable debugging by restarting Data-Protector daemons then do the same again. # cmhaltpkg <package_hostname> # cmrunpkg [-n <node_name>]<package_hostname> Collecting omniinet debugs Edit omni entry in /etc/inetd.conf Example omni stream tcp nowait root /opt/omni/lbin/inet inet -log /var/opt/omni/log/inet.log -debug 1-99,104-500 /tmpdir/DBG Get inetd to reread its new configuration inetd -c Debugs will be generated under /tmpdir/*DBG

Media Agent debugs

For Media Agent specific problems, the following general guidelines can be applied: 1-200 for most common cases

15

1-300 for SHMIPC 1-350 for consolidation 1-505 for memory tracking Note that ranges 220,221 are used for timing information. These ranges are hardcoded and hence not seen in the list of range definitions below. Include these ranges to get device timing information.

Here's a list of range definitions:

#define DBG_MA_OPTIONS #define DBG_MA_NETINIT #define DBG_MA_LIBDEV #define DBG_MA_LIBSCSI #define DBG_MA_AUX #define #define #define #define #define #define #define #define #define DBG_MA_WORK DBG_MA_WORK_LOW DBG_MA_CATALOG DBG_MA_LIBPOL DBG_MA_LIBMH

2 3 61 62 63 31 131 32 33 34

DBG_MA_STATES 11 DBG_MA_CONSOLIDATE 12 DBG_MA_CONSOLIDATE_LOW 112 DBG_MA_CONSOLIDATE_LOW_LOW 212 21 22 23 51

#define DBG_MA_CONMSG #define DBG_MA_NETMSG #define DBG_MA_NETEVENT #define DBG_MA_LIBCTRL

#define DBG_MA_UTIL 71 #define DBG_MA_UTIL_SIMULATE 72 #define DBG_MA_UTIL_SIMULATE_LOW 172 #define DBG_MA_SCSITAB_ERR #define DBG_MA_SCSITAB #define #define #define #define DBG_FTS DBG_FTS_LOW DBG_FTS_LOWLOW DBG_FTS_HASHTABLES 78 207 47 147 247 347

Information required for Cell Manager and RDS problems.

The following information should be obtained An exact, and detailed, description of what the problem is. Supply a copy of any error messages. When did the problem start? Details any change to the backup environment that may have caused the problem. Can the problem be reproduced at will? If the problem can be reproduced, include details of the exact steps that cause the problem. Explain how the customer expects DP to work when it is OK. If possible, run an omnidbcheck -extended` and supply the text output. If, for time reasons, an omnidbcheck -extended` is not possible, run `omnidbcheck` and supply the text output. If the customer has found a workaround for the problem, let us know what that is. If the problem is seen, or caused, using the GUI, include whether the GUI runs on the DP Cell Manager or another system. If another system, where possible, see if the problem also exists when using a GUI running on the Cell Manager. If the problem exists on a GUI running on another system, include details of that system (hostname, DP version and patch level, Operating System). Confirm if the IDB backup runs OK, or not. If they have problems with their IDB backup confirm when they last had a successful IDB backup. See Appendix A for a list of helpful data for RDS problems where DP runs on HPUX. See Appendix B for a list of helpful data for RDS problems where DP runs on Windows. Regarding the collection DP debug files for RDS crashing problems. If RDS is crashing intermittently, then usually DP debugs file collection is not required. If a specific operation is causing RDS to crash/hang, then debugs of that operation should be collected. Regarding the collection of the RDS debug file (RDSdebug.log). It is possible to put RDS in a Raw Logging mode by use of the rdmserver setting LogRawCoreAPI=1. This is not usually required in the initial data for an RDS problem. If it is required it will be requested by the Data Protector lab. Appendix A Data list for RDS problems on HP-UX Minimum data All /var/opt/omni/server/db40/datafiles/catalog/RDS* log files grep -v ^# /etc/opt/omni/server/options/global | grep -v "^ *$" > globalactivelines.txt omnicheck -patches > patchDP_check.txt tail -n 5000000 /var/opt/omni/log/debug.log > debug_tail.log Further Usual Data The data listed in the "Minimum data" section +... All /var/opt/omni/server/log/Check_* log files /var/opt/omni/server/db40/datafiles/catalog/rdmserver.ini file /var/opt/omni/server/log/HealthCheck.log file /opt/omni/.omnirc file grep -v ^# /opt/omni/.omnirc | grep -v "^ *$" > omnircactivelines.txt /etc/opt/omni/server/options/global file /etc/opt/omni/server/options/trace file

grep -v ^# /etc/opt/omni/server/options/trace | grep -v "^ *$" > traceactivelines.txt crontab -l > crontab_op.txt what /stand/vmunix > what_vmunix.txt uname -a > uname_a.txt model > model_out.txt swlist -l fileset | grep -i OV DP > patchDP_swlist.txt swlist -l product > swlist_prod.txt swlist -l fileset > swlist_file.txt swlist -l fileset -a state > swlist_a.txt tail -n 5000000 /var/opt/omni/log/inet.log > inet_tail.log omnicc -ver > omnicc_ver.txt sysdef > sysdef_out.txt kctune | egrep -i "maxdsiz|semmnu|maxssiz" > kctune_siz.txt kctune > kctunelist.txt bdf > bdf.txt du -skx /var/opt/omni/server/db40 > db40_du_skx.txt omnidbutil -info > omnidbutil_info.txt omnidbutil -extendinfo > omnidbutil_extendinfo.txt omnidbutil -list_large_directories 100000 > /list_dirs_100000.txt *** /etc/opt/omni/server/cell/cell_info file /etc/opt/omni/client/cell_server file ls -l /var/opt/omni/server/log > ls_varoptomniserverlog.txt ls -l /var/opt/omni/log > ls_varoptomnilog.txt ls -l /var/opt/omni/tmp/ > var_opt_omni_tmp.txt ps -ef | grep omni > ps_ef_omni.txt ps -ef > ps_ef.txt tail -n 5000000 /var/adm/syslog/syslog.log > syslog_tail.log omnirpt -report db_purge_preview > purge_preview.txt *** /opt/omni/sbin/utilns/get_info > get_info_op.txt *** Copy of the contents of dirs... /etc/opt/omni/server/rptgroups /etc/opt/omni/server/rptschedules If a core file exists in either /var/opt/omni/log or var/opt/omni/server/log file core text output If DP 6.2x please also collect omnicc -encryption -status all > omnicc_encry_status_all.txt Extra Data that may reduce the chance that we will ask for more. The data listed in the "Minimum data" and "Further Usual Data" sections above. tail -n 1000 /var/opt/omni/server/log/media.log > medialog_tail.txt omnidb -sess -last 20 > omnidb_sess_last_20.txt omnidownload -list_libraries > list_libs.txt omnidownload -list_devices > list_devs.txt omnidownload -list_devices -detail > list_devs_det.txt omnidownload -list_libraries -detail > list_libs_det.txt omnicellinfo -schinfo > omnicellinfo_schinfo.txt` omnidbutil -list_dcdirs > list_dcdirs.txt` omnidbutil -list_large_directories 500000 -top 50 -detail > list_dirs_500000.txt *** /var/opt/omni/server/db40/logfiles/rlog/obrindex.dat file omnicc -debug 20 SETTINGS.txt - makes a file in /tmp ending in SETTINGS.txt *** Note: These commands can put a load on DP, best run when cell is quiet. Appendix B Data list for RDS problems on Windows Note: The PATH values list are default PATHs for DP on Windows 2008.

Minimum data All RDS* files from C:\ProgramData\OmniBack\db40\datafiles\catlog\ C:\ProgramData\OmniBack\Config\Server\Options\global omnicheck -patches > patchDP_check.txt C:\ProgramData\OmniBack\log\debug.log Further Usual Data The data listed in the "Minimum data" section +... C:\ProgramData\OmniBack\log\server\log\Check_* log files C:\ProgramData\OmniBack\db40\datafiles\catlog\rdmserver.ini C:\ProgramData\OmniBack\log\server\log\HealthCheck.log file C:\ProgramData\OmniBack\omnirc file (if it exists) C:\ProgramData\OmniBack\Config\Server\Options\trace file C:\ProgramData\OmniBack\log\inet_tail.log file omnicc -ver > omnicc_ver.txt omnidbutil -info > omnidbutil_info.txt omnidbutil -extendinfo > omnidbutil_extendinfo.txt omnidbutil -list_large_directories 100000 > /list_dirs_100000.txt *** C:\ProgramData\OmniBack\Config\Server\cell\cell_info file omnirpt -report db_purge_preview > purge_preview.txt *** /opt/omni/sbin/utilns/get_info > get_info_op.txt *** Copy of the contents of dirs... C:\ProgramData\OmniBack\Config\Server\rptgroups C:\ProgramData\OmniBack\Config\Server\rptschedules dir listing of directory C:\ProgramData\OmniBack\tmp dir listing of directory C:\ProgramData\OmniBack\DB40 If DP 6.2x please also collect omnicc -encryption -status all > omnicc_encry_status_all.txt If the server is Windows 2003, the boot.ini file. The System and Event log files in text format. Extra Data that may reduce the chance that we will ask for more. The data listed in the "Minimum data" and "Further Usual Data" sections above. omnidb -sess -last 20 > omnidb_sess_last_20.txt omnidownload -list_libraries > list_libs.txt omnidownload -list_devices > list_devs.txt omnidownload -list_devices -detail > list_devs_det.txt omnidownload -list_libraries -detail > list_libs_det.txt omnicellinfo -schinfo > omnicellinfo_schinfo.txt` omnidbutil -list_dcdirs > list_dcdirs.txt` omnidbutil -list_large_directories 500000 -top 50 -detail > list_dirs_500000.txt *** If a process is crashing, sometimes it will be necessary to enable Dr. Watson logging and supply Dr. Watson logs of the failing process. *** Note: These commands can put a load on DP, best run when cell is quiet.

Information required for Disk and Media Agent Issues

Problem Description Clearly describe the specific issue to be addressed in the case. The problem description in the case needs to only describe a single problem. The description will need to include the following:

List the exact steps the customer performed which produced the undesired result. For example, specify the exact command, what buttons were selected in the GUI, and what was entered in any of the GUI fields. Describe exactly what you and the customer observed. Describe exactly what you and the customer expected. Specify when the problem first occurred. Specify what has changed in the environment that could have caused the problem. Describe, if possible, what you or the customer has determined as a workaround.

Collect Debugs Verify the latest Disk Agent, Media Agent and Core binaries have been properly installed on the problematic client. A patch list is welcome. Isolate the problem to as small of an area as possible. For example, if multiple hosts are involved in the issue, isolate the issue to a single host. If multiple volumes are involved, isolate the issue to a single volume. If multiple files are involved, isolate the issue to a single file. Reproduce the issue using the following debug ranges. 1-500 debugs for normal sessions 1-999 If the Disk Agent appears to be hanging, core-dumping, or abending (Netware). These can be circular debugs. Collect the debugs logs from the Cell Manager, Media Agent client(s) and problematic Disk Agent client(s). Verify the debug files are complete. The set will need to include disk agent, media agent, Session Manager (BSM, RSM, CSM & MSM), debug.log from each host and complete session report. Verify the debugs were created for the correct debug range. Verify the Disk Agent debugs are from the latest binary. Package the files in either in WinZip format or a standard format on HP-UX.

Collect Matching Session Report. From the command line use the command omnidb and direct the output of the session report which matches the debugs to a file named session.out. This will be an ascii text file. Note: Do not convert this file to a different file format. Collect session.out. Verify session.out has captured the issue which the WTEC case is opened for. Verify timestamps in the session report match the timestamps of the debugs logs. Use session.out to verify that debugs for every host reported has been accounted for.

Collect Supporting Data. In many cases it will become necessary to collect the following supporting information. Windows System: drwtsn32.log, System & Application Event Logs Note: Windows Event logs must be saved as text files only. Do not supply .EVT formatted event logs. Novell Netware: Config .txt file and Abend log file if the issue is for a Novell Netware abend.

Run the vbda as standalone command It is often interesting to run the vbda as a command. E.g. for HP-UX/ Linux # /opt/omni/lbin/vbda -vol /apps/all/dir -trees /apps/all/dir/admin out /dev/null -profile -debug 1-500,T:1000 /alternate/debug.txt Here debugs are created under the alternate directory.

Here are examples on Windows besides volumes as C:,D:,E:.. we have also /CONFIGURATION. In quick mode, without reading data: vbda -vol /CONFIGURATION -profile -preview In physical reading mode: vbda -vol /CONFIGURATION -profile -out nul In debug mode: vbda -vol /CONFIGURATION -profile -out nul -debug 1-200 out.txt In debug with reporting: vbda -vol /CONFIGURATION -profile -report 0 -out nul -debug 1-200 out.txt

Share with colleagues