Beruflich Dokumente
Kultur Dokumente
Author
Name
Project
Murali Thangavel
Applied Materials
Reviewed by
S.No.
1
2
3
4
5
6
7
8
Topics
Veritas Disk Headers
Veritas Configuration Database
Configuration Backup
Restoring configuration
Recovering from path failure
Recovering deleted volume
Veritas Explorer
Troubleshooting
log
priv 059891-068961[009071]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c1t3d0s2
state=enabled
#
Due to the critical nature of the configuration information stored in the disk
headers, it is important to periodically back up this information. As I will discuss in
the section Configuration Backups, Veritas introduced new features in Veritas
Volume Manager 4.X to ease the backup process.
To print the contents of the configuration database, the vxprint utility can be
used. The following example uses the vxprint "-h" (list record hierarchies) and "-t"
(use one-line format tailored for each record type) options to display the
configuration database with an informational header and descriptive records
bash-3.00# vxprint -ht -g datadg
DG NAME
NCONFIG
NLOG MINORS GROUP-ID
ST NAME
STATE
DM_CNT SPARE_CNT
APPVOL_CNT
DM NAME
DEVICE
TYPE PRIVLEN PUBLEN STATE
RV NAME
RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME
RVG
KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME
CACHEVOL KSTATE STATE
VT NAME
RVG
KSTATE STATE NVOLUME
V NAME
RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME
VOLUME
KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME
PLEX
DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME
PLEX
VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME
PLEX
CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME
PARENTVOL LOGVOL
SP NAME
SNAPVOL
DCO
EX NAME
ASSOC
VC
PERMS MODE STATE
SR NAME
KSTATE
dg datadg
default
dm disk01
dm disk02
c1t2d0s2
c1t3d0s2
v vbr
default 35000
auto
auto
81151
81151
1265408259.16.ossa
286596864 286596864 -
fsgen
pl vbr-01
vbr
ENABLED ACTIVE 20971520 CONCAT
sd disk01-01 vbr-01
disk01 0
20971520 0
c1t2d0
pl vbr-02
vbr
ENABLED ACTIVE 20971520 CONCAT
sd disk02-01 vbr-02
disk02 0
20971520 0
c1t3d0
bash-3.00#
RW
ENA
RW
ENA
The object types (e.g., subdisks, disk groups, plexes, volumes, etc.), block offsets,
object names, and locations can be useful for understanding the Veritas Volume
Manager layout and reassembling the configuration layout during a disaster. Due
to the importance of this information, backing up the configuration database is
imperative! As you will see in the next section, Veritas Volume Manager makes
backing up the configuration information a snap.
Configuration Backups
The vxconfigbackupd daemon was introduced in Veritas Volume Manager 4.X
to provide an automated way to back up disk headers and the configuration
database. Vxconfigbackupd is started automatically at system boot time and
actively monitors the configuration database and disk headers. When
vxconfigbackupd detects a change to the active configuration,
vxconfigbackupd will write the configuration to a set of files in the
/etc/vx/cbr/bk/<dgname>.<dgid> directory. ("<dgname>" identifies the disk
group, and "<dgid>" identifies the disk group id.) The vxconfigbackupd man
page provides the following descriptions for the files in the
/etc/vx/cbr/bk/<dgname>.<dgid> directory:
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.diskinfo
Location of backup file for disk attributes.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.binconfig
Location of backup file for binary configuration copy.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.cfgrec
Location of backup file for configuration records in vxprint -m format.
In addition to the automated backups that are performed by
vxconfigbackupd, manual backups can be performed with the
vxconfigbackup utility. This is valuable for backing up specific versions of the
configuration and archiving the configuration to a user-supplied destination.
The following example uses the vxconfigbackup "-l" (location to store backup)
to back up the current configuration to the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups
Start backing up diskgroup oradg to \
/etc/dgbackups/oradg.1127240283.19.ossa ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: oradg
If you are using a version of Veritas Volume Manager prior to 4.X, you will be
unable to use the vxconfigbackupd and vxconfigbackup utilities to manage
backups. Don't fear -- the sample shell script in Listing 1 can be used to archive
the configuration to a text file, which can be passed to vxmake "-d" option to
re-create the configuration if disaster were to strike.
Veritas Volume Manager (Recovery techniques and Troubleshooting)
Restoring Configuration
When configuration information is lost due to a system outage or a mistake at
the keyboard, the configuration can often be recovered to a point in time
(preferably to a point in time when a known working backup was taken with
vxconfigbackup ) with the vxconfigrestore utility. The following example shows
how to restore the contents on the oradg disk group using the configuration
backup in the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups oradg
Diskgroup oradg configuration restoration started ......
Run:
vxconfigrestore -l /etc/dgbackups/ -c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/ -d oradg ==> to abort the restoration.
When vxconfigrestore is invoked, the configuration will be staged to a transient
location, allowing the administrator to validate the configuration with the
vxprint utility. If the configuration looks sound, the configuration can be
committed to the disk headers and the configuration database by invoking
vxconfigrestore with the "-c" (commit) option:
Veritas Volume Manager (Recovery techniques and Troubleshooting)
The Veritas vxprint utility was also unable to display the disk group
configuration (since the configuration database was unavailable):
$ vxprint -g oradg -ht
$
10
Got the information from HnE, that the storage array was powered off, Based
on their findings that a power cords failed. Once they replaced the cord, the
storage array came back to life, but the oradg disk group I needed to access
was in the disabled state:
$ vxdg list
NAME
STATE
oradg
disabled
ID
1123603158.13. ossa
A quick check of the disks showed that they were online and associated with
the disabled disk group:
$ vxdisk list
DEVICE
TYPE
DISK
GROUP
STATUS
c0t0d0s2
auto:none
online invalid
c0t1d0s2
auto:none
online invalid
c1t1d0s2
auto:cdsdisk
c1t1d0
oradg
online dgdisabled
c1t2d0s2
auto:cdsdisk
c1t2d0
oradg
online dgdisabled
c1t3d0s2
auto:cdsdisk
c1t3d0
oradg
online dgdisabled
c1t4d0s2
auto:cdsdisk
c1t4d0
oradg
online dgdisabled
c1t5d0s2
auto:cdsdisk
c1t5d0
oradg
online dgdisabled
c1t6d0s2
auto:cdsdisk
c1t6d0
oradg
online dgdisabled
In some situations, Veritas may report offline devices as "failed was: cXtXdXs2".
When this happens, the vxreattach command can reconnect Veritas Volume
Manager to "lost" devices. Luckily, in our case, Veritas was able to reconnect
to the devices so I unmounted the file system, deported the disk group, and
imported the disk group to enable the oradg disk group:
$ umount /oracle/admin
$ vxdg deport oradg
$ vxdg import oradg
11
Deport and Import operations are required to fix a disabled disk group and will
validate that the disk group configuration records are consistent. Once the
disk group was imported, I ran vxinfo to view the volume and plex status.
$ vxinfo -g oradg p
vol oravol01
fsgen Startable
plex oravol01-03 ACTIVE
vol oravol01-L01 fsgen Startable
plex oravol01-P01 ACTIVE
plex oravol01-P02 ACTIVE
vol oravol01-L02 fsgen Startable
plex oravol01-P03 ACTIVE
plex oravol01-P04 ACTIVE
vol oravol01-L03 fsgen Startable
plex oravol01-P05 ACTIVE
plex oravol01-P06 ACTIVE
The "Startable" flag indicates that the volume and plexes are in a startable
state, so I executed the vxvol utility to start the volume:
$ vxvol -g oradg start oravol01
Once the volume came online, I ran fsck to replay the transactions in the VxFS
journal:
$ fsck -F vxfs /dev/vx/dsk/oof/oravol01
log replay in progress
replay complete - marking super-block as CLEAN
After fsck finished the consistency check, I mounted the file system. Due to the
recovery features built into Veritas volume manager and Veritas file system, we
were able to avoid a full file system restore! Since I received the failure
notification immediately after Veritas detected a problem with the volume, I
was able to reduce the time it took to recover the faulted system.
12
Run:
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -c oradg ==> to commit
the restoration.
13
NLOG
MINORS GROUP-ID
DM_CNT SPARE_CNT
DM NAME DEVICE
TYPE
KSTATE STATE
CO NAME CACHEVOL
VT NAME NVOLUME
REM_HOST REM_DG
REM_RLNK
KSTATE STATE
KSTATE STATE
APPVOL_CNT
KSTATE STATE
LENGTH LAYOUT
NCOL/WID MODE
SD NAME PLEX
DISK
SV NAME PLEX
SC NAME PLEX
MODE
DCO
14
dg oradg
default
default 10000
1127240283.19. ossa
oravol01-03 fsgen
2/2
ENA
2/2
ENA
2/2
ENA
..
.
v oravol01-L03 -
fsgen
oravol01-P05 c1t3d0 0
13981056 0
c1t3d0 ENA
oravol01-P06 c1t6d0 0
13981056 0
RW
RW
c1t6d0 ENA
To ensure that no damage had occurred to the file system, we used fsck to
consistency check the file system:
$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01
and got our Oracle DBA to verify the integrity of each data file with Oracle's
database verification utilities. Because accidents and disasters can strike at
any time, it is important to perform regular backups of the Veritas
configuration and store this data in a safe.
15
Veritas Explorer
VRTSexplorer , also known as Veritas Explorer, VxExplorer, vxexplorer, and
vxexplore, is a tool provided by Symantec Corporation Technical Support that is
executed on a host to gather some of the information that may be needed by a
Support Engineer to troubleshoot an issue. The output of this utility is a compressed
tar file that must be sent to Support. The information collected may not be
sufficient to resolve the issue, but it is likely to give very relevant information about
it.
The use of VRTSexplorer utility involves three different steps:
1.
2.
3.
Obtaining VRTSexplorer
Executing the VRTSexplorer binary
Sending the output to Symantec Technical Support (Vendor)
16
220
Name (ftp.veritas.com <ftp://ftp.veritas.com/>): anonymous
The below message appears with a prompt for a password. Use a full email
address as the password:
331 Guest login ok, send your complete e-mail address as password.
Password: customer@email.com <mailto:customer@email.com>
While logged into the FTP server as the user anonymous, it is not possible to
list any files or directories. It is important that the below steps are followed
exactly:
Change directories to /pub/support:
ftp> cd /pub/support
Set FTP mode to binary:
ftp> bin
Download the file that contains utility:
ftp> get vxexplore.tar.Z
Close the ftp connection:
ftp> bye
Untar the download:
Untar the downloaded file (only if the VRTSexplorer utility has been
downloaded and not installed with the VRTSspt package) :
# zcat vxexplore.tar.Z | tar xvf The above command will create a VRTSexplorer directory containing
everything that is needed. Change into this directory:
# cd VRTSexplorer
2. Execute the VRTSexplorer utility:
Some notes on the execution of VRTSexplorer:
To run the VRTSexplorer utility for only one particular product, such as
Veritas Volume Manager, execute:
# ./VRTSexplorer vxvm
17
Refer to the README file for additional options that are available with the
VRTSexplorer utility.
The below messages will appear requesting input for the case number,
destination directory, restart of vxconfigd (do not choose to do this if
PowerPath is installed or if this is a node in a cluster), etc. The program will
output several messages and finish by tarring and compressing the
collected information. Please note that this operation can take some time
to complete.
Below is partial example of what might be seen (the messages below are
from VRTSexplorer version 5.5o and will be different from earlier versions of
the utility) when executing the VRTSexplorer utility to gather information
about a host, output is dependant on which product packages are
installed:
# ./VRTSexplorer
VRTSexplorer: Initializing.
VRTSexplorer: Please enter case number, or just hit enter: 150-175-206
VRTSexplorer: Please select a destination directory (default: /tmp): /tmp
VRTSexplorer: Using /tmp as destination directory.
VRTSexplorer: Collecting system configuration information for SunOS system.
VRTSexplorer: Collecting VERITAS package version information.
VRTSexplorer: Collecting loadable module information.
VRTSexplorer: Collecting SLIM information.
VRTSexplorer: Collecting SLIM Agent installer logs.
VRTSexplorer: Collecting SLIM Server installer logs.
VRTSexplorer: Collecting Cross Product/Platform Installation (CPI)
information.
VRTSexplorer: Collecting ISIS configuration information.
VRTSexplorer: Collecting VERITAS SANPoint Control Console configuration
information.
##### All information and files will be placed under
/tmp/VRTSexplorer_150-175-206_server1/spc
Logfile is /tmp/VRTSexplorer_150-175-206_server1/spc/LOG
##### Capturing SAL/VAIL/VEA information from host server1'
##### Saving VERITAS Package Information..........
##### Logging VEA Information #####
+Copying vxisis.log files.
+Copying /var/vx/isis vxsvc corefiles
+Copying /opt/VRTSob/bin vxsvc corefiles
##### Logging VRTSvail Information #####
Checksum /opt/VRTSvail/providers/vx*:...............
Veritas Volume Manager (Recovery techniques and Troubleshooting)
18
19
20
Troubleshooting
-Troubleshooting the Boot Process
-Recovering the Boot Disk Group
Files Used in the Boot Process
* /etc/system
Contains VxVM entries
* /etc/vfstab
Maps mount points to devices
* /etc/vx/volboot
Contains disk ownership data
* /etc/vx/licenses/lic, /etc/vx/elm
Contains license files
* /var/vxvm/tempdb
Stores data about diskgroups
* /etc/vx/reconfig.d/state.d/install-db
Indicates VxVM is not initialized
* /VXVM#.#.#-UPGRADE/.start_runed
Indicates that the VxVM upgrade is not complete
21
22
23
24
Conclusion
This document provided an introduction to important Veritas Volume Manager
recovery techniques and discussed ways to ensure recoverability after a
disaster. When dealing with difficult Veritas situations, it is imperative to know
what function a command performs before executing it. It is also helpful to
engage Veritas support or the folks on the Veritas mailing lists when major
disasters occur. This will ensure that a second set of eyes is available to review
the problem, and will ensure the quickest and safest methods are used to
recover a faulted system. I tested all commands on a Solaris 10 server running
Veritas Volume Manager 4.1, and I highly recommend testing all configuration
changes on non-production systems.
25