VERITAS Confidential

Welcome
To
VERITAS Presentation
 Volume Manager for Solaris

VERITAS Confidential
 Suite of Products

• VERITAS Volume Manager (VM)

• VERITAS FlashSnap
• Dynamic Multipathing Software (DMP)
• VERITAS Volume Replicator (VVR) Support
• VERITAS Cluster Server (VCS) Support
• Microsoft Cluster Service (MSCS) Support

VERITAS Confidential
 VERITAS Volume Manager
General features are:
• Centralized Storage Management across
Entire Domain
– Supports concurrent local and remote client
connections to the server for browsing and
update capabilities; thus, it’s possible to
administer online storage remotely.
– Provides asynchronous notification to all
connected clients whenever any change to
the storage configuration takes place.

VERITAS Confidential
 VM – General Features
• Configuration Solutions
– Ability to configure and manage different
volume layouts: concatenated, striped,
mirrored, mirrored striped, and RAID-5
volumes. Supports up to 32-way mirrors on a
mirrored volume.
– Automatic detection of failed disks and the
ability to repair fault-tolerant volumes on those
disks without interrupting applications using
the volumes.

VERITAS Confidential
 VM – General Features
- Provides storage migration path with the ability
to upgrade existing Windows NT 4 or “basic
disk” partitions to the new “dynamic” volumes
based on VERITAS Volume Manager
technology.
- Supports and manages FT (fault-tolerant)
disks and volumes previously created in a
Windows NT system with Disk Administrator.
- Supports online extending of all volume types.

VERITAS Confidential
 VM – General Features
- Supports a cluster environment that runs
under VERITAS Cluster Server software or
Microsoft Cluster Server software.
- Supports mounting a volume without requiring
the use of a drive letter.(windows only)
- Supports moving of storage between
computers with the Import and Deport
Dynamic Disk Group functions.
- Command line support for administrators who
prefer this method over a GUI.

VERITAS Confidential
 VM – General Features
• Optimized System Performance
- I/O statistics are used to identify high-traffic
areas, known as “hot spots.” You can use the
Volume Manager Move Subdisk command to
resolve these hot spots online.
- Data is assigned to physical drives to evenly
balance the I/O load among the disk drives.
This is known as storage load balancing.
- Event logging of errors and important
information is provided.

VERITAS Confidential
 GUI vs. Command Line

GUI Command Line

• Easy to use • Lots of options

• Multi Platform • Works remotely from
• Makes hard tasks easy nearly any place.
• Easy error detection • Makes very hard tasks
• Alert logs possible.
• Steep learning curve

VERITAS Confidential
 VERITAS Enterprise Administrator
VERITAS Enterprise Administrator (VEA), provides
access to applications on managed hosts.

VERITAS Confidential
 Storage Objects
• Disk Group
– A set of disks that are an administrative domain for
creating and managing volumes. Volumes can’t span
disk groups.
– Each disk stores a copy of the configuration data for
the whole group. Dividing all disks into smaller
groups reduces the amount of configuration data
that have to be stored on each disk.
• Disk
– Volume Manager disk objects persist even when the
physical disk they represent are removed from the
system.
VERITAS Confidential
 VEA – Disk View tab

VERITAS Confidential
 Storage Objects
• Volume
– A virtual disk without the size limitations, and with
better performance and reliability than the
underlying physical disks.
• Subdisk
– A contiguous region on a disk that is part of the
physical storage for a volume. Subdisks can’t
overlap.
• Plex
– A component of a volume that contains one
complete copy of the data or a log.

VERITAS Confidential
 VEA – Volume tab

VERITAS Confidential
 VEA – Subdisks and Plex

VERITAS Confidential
 Storage Object Relationships in General

Disk group
1
1
N
N

Disk Volume
1
1 1

N 1

Plex File System

N N

Subdisk

VERITAS Confidential
 VRTSexplorer
• Allows support to check the health of the
System

•Provides us with the understanding of the

system, so that support can make right
recommendation

•Gives support a true glance of the problem or

issue being resolved.
VERITAS Confidential
 How to get VERITAS
VRTSexplorer
• # ftp ftp.veritas.com • VERITAS Volume Manager 3.5 “Ships with
vxexplorer and other tools”
• # Login: anonymous • Trouble Shooting Tools are located ins
• # Passwd: <email address> package “VRTSspt”
• # cd /pub/support
• # bin
• # get vxexplore.tar.Z

• In Windows Explorer
• ftp://ftp.veritas.com/pub/support/vxexplor
e.tar.Z

VERITAS Confidential
 How to Run VRTSexplorer
#cd VRTSspt
#cd VRTSexplorer
./VRTSexplorer
VRTSexplorer: Initializing.
VRTSexplorer: Please enter case number, or just hit enter:100000001
VRTSexplorer: Please select a destination directory (default: /tmp):
VRTSexplorer: Collecting system configuration information for SunOS system.
VRTSexplorer: Collecting VERITAS package version information.
VRTSexplorer: Collecting loadable module information.
VRTSexplorer: Collecting ISIS configuration information.
VRTSexplorer: Collecting SIG licensing information.
VRTSexplorer: Determining current VxVM operating mode.
VRTSexplorer: Collecting VxVM configuration information.
VRTSexplorer: Collecting DMP configuration information.
VERITAS Confidential
NOTICE: This section will stop and restart the VxVM Configuration Daemon,
vxconfigd. This may cause your VxVA, VMSA and/or VEA session to exit.
This may also cause a momentary stoppage of any VxVM configuration
actions. This should not harm any data; however, it may cause some
configuration operations (e.g. moving subdisks, plex
resynchronization) to abort unexpectedly. Any VxVM configuration
changes should be completed before running this section.

If you are using EMC PowerPath devices with VERITAS Volume Manager,
you must run the EMC command(s) 'powervxvm setup' (or 'safevxvm
setup') and/or 'powervxvm online' (or 'safevxvm online') if this
script terminates abnormally.

Restart VxVM Configuration Daemon? [y,n] (default: n)y

VRTSexplorer: Collecting VRAS configuration information.
VRTSexplorer: Script finished.
VRTSexplorer: Please ftp /tmp/VRTSexplorer_100000001_aisia.tar.Z to ftp.veritas.com:/incoming
 What is in VRTSexplorer

arp_a ifconfig_a nfs uname_a

cron iostat_En pam uptime
dev isainfo_v pkginfo usr
df_klaV isis pkginfo_l var
df_klag kernel pkgparam_patchlist vexplore_version

df_klat license prtconf vmstat

eeprom modinfo prtdiag vmstat_s
env mount_v ps_elf vras
etc netstat_i reboot vxfs
hosted netstat_r showrev vxvm---Directory

VERITAS Confidential
 VXVM Directory

Dmp-----Directory vxprint_ht
vxdctl_c_mode vxprint_m_debdg
vxdctl_mode vxprint_m_deq
vxdg_list vxprint_m_dg1
vxdg_list_debdg vxprint_m_rootdg
vxdg_list_deq vxprint_mpvshr_debdg
vxdg_list_dg1 vxprint_mpvshr_deq
vxdg_list_rootdg vxprint_mpvshr_dg1
vxprint_mpvshr_rootdg
vxdisk_list
vxstat_g_debdg
vxdisk_list_c0t0d0s2
vxstat_g_deq
For Every Disk vxstat_g_dg1
vxdisk_s_list vxstat_g_rootdg
vxkprint
vxprint
VERITAS Confidential
 Common Issues on Solaris

DMP
Device Failures
Package Install/Upgrade
Disaster Recovery
(Duplicate Disk Group Copies)

VERITAS Confidential
 DMP
• DMP Functionality
• When is DMP installed
• DMP Flow Chart
• DMP Error Handling
– DMP Supported Arrays (Appendix A)
– Array Types (Appendix B)

VERITAS Confidential
 DMP Functionality

• Provide a single path abstraction from each disk

device to VxVM
– Handles I/O across these paths
• Load balancing of I/Os across the multiple path
– Feature currently available only for Active/Active(A/A) DMP
Arrays
• Transparent fail over from one path to another

VERITAS Confidential
 DMP Flow Chart

VERITAS Confidential
 When is DMP installed
• The DMP driver is installed during the VxVM
pkgadd
• This can be verified by doing looking at the
modinfo file in the explorer
– #more modinfo | grep VxVM
• 23 10202043 1188e8 263 1 vxio (VxVM 3.5s_p2.8 I/O driver)
• 25 10305b3e 1b23c 262 1 vxdmp (VxVM 3.5s_p2.8 DMP Driver)
• 26 1031e81a 80f 264 1 vxspec (VxVM 3.5s_p2.8 control/status)

VERITAS Confidential
 Error Handling in DMP
• Error daemon is awakened whenever I/O returns
with an error

• Error daemon then checks the status of the path

in a disk array by doing SCSI inquiry on the path

• If path is not alive, error daemon disables the

path and retries the I/O on the other path before
returning error

VERITAS Confidential
 Error Handling in DMP(contd)

• If path is alive, the error daemon retries the I/O,

a default of five times. If I/O fails even after the
retries DMP assumes it is a media

VERITAS Confidential
 Disk Failure
• Determine the disk(s) failing
– Use command vxdisk list

• Determine the disk group the failure is involved

with
• Check the volume,plexes and sub-disk states
– Use command vxprint –ht

• Check message log to determine cause of failure

VERITAS Confidential
 vxdisk list

DEVICE TYPE DISK GROUP STATUS

c1t13d0s2 sliced testdg04 testdg online
c1t14d0s2 sliced testdg03 testdg online
c1t15d0s2 sliced - - error
c2t10d0s2 sliced rootdisk rootdg online
c2t13d0s2 sliced testdg01 testdg online
- - testdg00 testdg failed was:c1t15d0s2

VERITAS Confidential
 vxprint –g testdg -ht

v vol02 - ENABLED ACTIVE 2097216 RAID - raid5

pl vol02-01 vol02 ENABLED ACTIVE 2098272 RAID 4/32 RW
sd testdg04-02 vol02-01 testdg04 524356 699426 0/0 c1t13d0 ENA
sd testdg03-02 vol02-01 testdg03 524356 699426 1/0 c1t14d0 ENA
sd testdg00-02 vol02-01 testdg00 524356 699426 2/0 - NDEV
sd testdg01-02 vol02-01 testdg01 524356 699426 3/0 c2t13d0 ENA

VERITAS Confidential
/var/adm/messages
• Sep 16 13:19:34 aisia disk not responding to selection
• Sep 16 13:19:34 aisia vxdmp: [ID 997040 kern.notice] NOTICE:
vxvm:vxdmp: disabled path 32/0x160 belonging to the
dmpnode 239/0x20
• Sep 16 13:19:34 aisia vxdmp: [ID 148046 kern.notice] NOTICE:
vxvm:vxdmp: disabled dmpnode 239/0x20
• Sep 16 13:19:48 aisia vxio: [ID 686135 kern.warning]
WARNING: vxvm:vxio: object testdg00-02 detached from RAID-
5 vol02 at column 2 offset 0
• Sep 16 13:19:48 aisia vxio: [ID 354480 kern.warning]
WARNING: vxvm:vxio: RAID-5 vol02 entering degraded mode
operation

0x20 = 32 Decimal ---This is the DMP node made by Veritas DMP to

match the node you must convert the Hex number
/dev/vx/rdmp & dmp
brw------- 1 root other 239, 32 Sep 14 21:18 c1t15d0s0
 With VERITAS VRTSexplorer review
the following

• /vxvm/vxprint_ht
• /vxvm/vxdisk_list
• /var/adm/messages
– search for degraded

• Solution (Appendix E)

VERITAS Confidential
 Package Installation/Upgrade

• Make sure packages are installed completely

– VXVM requires packages to be installed in correct order
– If unsure of what order refer to the manual
– VXVM3.5 provides the (installer) script to assist in installation
only

• Check modules are correctly loaded for the Solaris

Version loaded on the system

• Check license keys are correctly listed

VERITAS Confidential
 pkginfo –l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 3.5,REV=06.21.2002.23.14
BASEDIR: /
VENDOR: VERITAS Software
DESC: Virtual Disk Subsystem pkgadd -d /cdrom/CD_name/volume_manager3.5/pkgs
PSTAMP: VERITAS-3.5s:21-Jun-2002
1-VRTSvlic
INSTDATE: Aug 20 2003 10:47
2-VRTSvxvm
HOTLINE: 800-342-0652 3-VRTSvmdoc
EMAIL: support@veritas.com 4-VRTSvmman
STATUS: completely installed 5-VRTSob
6-VRTSobgui
FILES: 603 installed pathnames 7-VRTSvmpro
22 shared pathnames 8-VRTSfspro
9 linked files
76 directories
340 executables
158087 blocks used (approx)

VERITAS Confidential
 /uname_a
SunOS aisia 5.9 Generic_112233-03 sun4u sparc SUNW,Sun-Blade-1000

/kernel
# more /kernel/drv/sparcv9/ls_l_vx
-rw-r--r-- 1 root sys 435312 Jun 21 2002 /kernel/drv/sparcv9/vxdmp
-rw-r--r-- 1 root sys 412304 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.7
-rw-r--r-- 1 root sys 427536 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.8
-rw-r--r-- 1 root other 435312 Aug 20 10:48 /kernel/drv/sparcv9/vxdmp.SunOS_5.9
-rw-r--r-- 1 root sys 2953480 Jun 21 2002 /kernel/drv/sparcv9/vxio
-rw-r--r-- 1 root sys 2866112 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.7
-rw-r--r-- 1 root sys 2908152 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.8
-rw-r--r-- 1 root other 2953480 Aug 20 10:48 /kernel/drv/sparcv9/vxio.SunOS_5.9
-rw-r--r-- 1 root sys 15336 Oct 31 2002 /kernel/drv/sparcv9/vxportal
-rw-r--r-- 1 root sys 19216 Jun 21 2002 /kernel/drv/sparcv9/vxspec
-rw-r--r-- 1 root sys 17840 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.7
-rw-r--r-- 1 root sys 18536 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.8
-rw-r--r-- 1 root other 19216 Aug 20 10:48 /kernel/drv/sparcv9/vxspec.SunOS_5.9

VERITAS Confidential
 /opt/VRTSvlic/bin
#./vxlicrep -e
License Key =
528222690330627075168549872296
Product Name = VERITAS Volume Manager
Lic Key Lib Ver = ELM
Key = Valid
License Type = PERMANENT_NODE_LOCK
Node Lock Type = 3 (Hostid and Architecture ID)

Features :=
RAID = Enabled

VERITAS Confidential
 Disaster Recovery
• Disaster Recovery(DR)
– Any event that will require a rebuild of a Disk Group
or Volume configuration database.
• Example Errors:
Duplicate Disk Group Records
Corrupted Disk Group Configuration
No Valid Disk Group Configuration Found

• When preparing for a disaster it is always a good plan to

have the following:

VERITAS Confidential
 Disaster Recover (contd)
• Layouts of the Diskgroup and Volumes
– vxprint –ht or vxprint -htr
– vxprint –g <diskgroup> -mpvshr

• If the above outputs cannot be provided due to a

“duplicate disk record” error. It is important to gather
the following before making any volume record
changes. (Example in: Appendix F)
– /etc/vx/diag.d/vxprivutil dumpconfig
/dev/rdsk/c#t#d#s<private> > dump.out
VERITAS Confidential
 Tools for Windows

• VxExplorer – gathers most of the necessary

information to troubleshoot a problem.
• Vxdisksetid – used to verify diskset id
consistency.
• Vxtool – advanced tools used by support and
engineering to diagnose and correct problems.

VERITAS Confidential
 Disk Group Configuration

• VxConfig stores a copy of a disk group’s

configuration on each disk in the group.
• A disk group’s configuration is described in
terms of configuration records.
• There is a record type for each object type.

VERITAS Confidential
 Configuration Record Types
• The types of configuration records
correspond to the VM object model
– Disk group record
– Volume record
– Plex record
– Subdisk record
– Two kinds of disk records

VERITAS Confidential
 Disk Media v.s. Disk Access Records
• Disk Media Configuration Record
– Identifies a disk by disk id and gives it an
administrative name, e.g. “disk1”.
– A disk group contains dm recs for all member disks.
– Subdisk recs are associated with dm recs.
– Dm recs can be disassociated from a failed physical
disk and re-associated with a replacement disk.
• Disk Access Record
– Identifies an O/S access path to a disk,
e.g. \device\Harddisk0.
– Not stored in the on-disk configuration database
because it can easily change after rebooting.
VERITAS Confidential
 Missing Disks

• If a disk group contains a disk media record for

a disk that Windows doesn’t know about (i.e. no
disk access record), the disk is a Missing Disk.

VERITAS Confidential
 Missing Disk

VERITAS Confidential
 VM Alert Message
Alert Message

VERITAS Confidential
 Event Viewer Alert

VERITAS Confidential
 Replace Disk

VERITAS Confidential
 Foreign Disks
• A disk is foreign if the hostid in it’s
configuration doesn’t match the server,

or

• If it’s disk group id matches an existing

disk group but it’s diskset id doesn’t

VERITAS Confidential
 Foreign Disk

VERITAS Confidential
 Merge Foreign

VERITAS Confidential
 Dynamic Disk Header

• A disk header records the disk’s VM disk

id, identifies the disk’s disk group, and
describes the layout of the private region.

VERITAS Confidential
 Disk Header Contents
• diskid: 8998ae08-f2a6-4264-97dc-8f5ceeda43de
• group: name=NewGrp id=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9
• flags: private autoimport
• import: bootsig=130 hostid=d0120cc1-8d21-11d7-aade-806d6172696f
• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe
• version: 2.12
• iosize: 512
• public: slice=0 offset=63 len=35551782
• private: slice=0 offset=35564430 len=2048
• update: seqno=0.36 time=1635821074
• headers: 2047 1856
• configs: count=1 len=1481
• logs: count=1 len=224

VERITAS Confidential
 Configuration Records
• Disk: Disk2 rid=0.1029 updated=0.1155
• assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de

• Group: NewGrp rid=0.1025 version=30 update=0.1154

• id: dgid=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9
• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe
•
• Volume: volB rid=0.1054 update=0.1151 mountname=F:
• info: len=1536000 guid=dda5ab3f-ac13-4915-8353-51d1a3615fd9
• state: state=ACTIVE
•
• Plex: volB-01 rid=0.1056 update=0.1151
• type: layout=STRIPE columns=3 width=128
• state: state=ACTIVE
•
• Subdisk: Disk2-01 rid=0.1049 updated=0.1155
• info: disk=0.1029 offset=0 len=256000 hidden=0
• assoc: plex=0.1041 (column=3 offset=0)
VERITAS Confidential
 Diskset id mismatch

• Diskset id mismatch means the disk was

removed from the disk group while the
disk was missing.

VERITAS Confidential
 Disketset Id
• Diskset id is used to make sure that a disk that’s
removed and then reinserted is not modified.
• A new diskset id is created when a disk is
missing.
• Diskset id is stored in the disk header and also in
the dg record in the config database.
• Each dm record also keeps track of the last
diskset id when its da is detached.

VERITAS Confidential
 Diskset Id Checking
• A disk cannot be onlined if the diskset id does
not match in its header and dg record in config.
• A disk can be brought online automatically in
rescan operation if the diskset id in the header
matches the “lastset” id in a detached dm record.

Disk: Disk2 rid=0.1029 updated=0.1158

assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de detached
lastset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe

VERITAS Confidential
 Vxdisksetid output

harddisk4
header diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7
config diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7

harddisk5 has a mismatched diskset ID

header diskset ID=b45dd49a-10aa-43f8-aa21-7c3fc4b3e1ff
config diskset ID=98a808a8-07cf-4175-8334-3419ed27b396

VERITAS Confidential
 Vxtool
vxtool - VERITAS Volume Manager diagnostic operations

Usage:
• vxtool disk <sub-operation> [-<options>] [<arguments>]
• vxtool kernel <sub-operation> [-<options>] [<arguments>]
• vxtool verify [-v] <volume-name> [[<offset>] <length>]
• vxtool getioparms <driveletter>:

VERITAS Confidential
 Vxtool disk setheader
Fields that can be modified:
• Diskid
• Hostid
• Dgid
• Dg_name
• Disksetid
• newdisksetid

VERITAS Confidential
 Vxtool disk setheader

The below command sets the diskset id entry in the header

record to match that of the config record. After a rescan the
disk can be brought back into the diskgroup successfully.

vxtool disk setheader harddisk5 diskset=98a808a8-07cf-4175-8334-3419ed27b396

VERITAS Confidential
 DMP Supported Arrays Appendix A

ARRAY TYPE VID PID A/A NEC DS3001, DS3002,

A/A SUN AP_NODES DS1000, DS1000F,
DS1100,DS1100F, DS3011,
A/A VERITAS DS1230, DS450, DS450F,
ATFNODES iStorage 1000, iStorage 2000,
A/AECCS all iStorage 4000
A/A EMC SYMMETRIX A/P SUN T300
A/A FUJITSU GR710, GR720, A/A VERITAS
GR730, GR740, GR820, RDACNODES
GR840 A/A SENA all
A/A HITACHI OPEN-* A/AIBM 2105
A/PG HITACHI DF350,DF400, A/ASSA SSA
DF500 A/A StorComp OmniForce
A/P LSI INF-01-00 A/PF VERITAS all
A/A NEC DS1200, DS1200F, A/A HP OPEN-*
DS3000SL, DS3000SM,
DS3001

VERITAS Confidential
 Array Types Appendix B

• A/A •Active/Active disk array; I/Os could be simultaneously active on more than one
path to the device without any performance degradation

• A/P •Active/Passive disk array; I/Os can not be issued to more than one path to the
device without severe performance penalty; when such arrays are con.figured in
auto-trespass mode,the path failover occurs with a normal I/O command to the
passive (standby) path
• A/PF
•Active/Passive disk array in explicit failover mode; A/P array that will require special
failover command; support for such arrays can not be added dynamically
• A/PG

•Active/Passive disk array with LUN group failover. A special case of A/P array
where a group of LUN’s fail-over

VERITAS Confidential
 vxdg_list Appendix C

VERITAS Confidential
 vxdisk_list_c1t15d0s2 Appendix D

VERITAS Confidential
 Solution Appendix E

• Vxdiskadm (Appendix G)
– Option 4 Remove a disk for replacement
– Option 5 Replace a failed or removed disk
• Volume should start rebuilding parity on new
device
– Use vxtask list to view status
• Example:
vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
171 171 R5RECOVER/R 32.09% 0/699426/224448
RECOV vol02 testdg00-02

VERITAS Confidential
 vxprivutil Appendix F

• Appendix C shows what disk have enabled

configuration copies
Example:
./vxprivutil dumpconfig /dev/rdsk/c2t15d0s3 >
/tmp/dump.out
• Appendix D show what slice the private
region is in.
• Verify the dump.out file with the following:
cat /tmp/dump.out | vxprint –D - -ht

VERITAS Confidential
 vxdiskadm Appendix G

Volume Manager Support Operations

Menu: Volume Manager/Disk

1 Add or initialize one or more disks

2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk

SO ON……….

VERITAS Confidential