You are on page 1of 24

This module focuses on the monitoring of Virtual Provisioning pools.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 1
One way to monitor Thin pool space is to run the symcfg monitor command with the i
option. This command runs every i seconds and permits the specification of a user defined
action script once a pool threshold has been reached. The action script can be set to execute
each time the condition is encountered, e.g. a 60% pool full condition or only once. In the
latter event the norepeat option causes the action script to execute only once.
The action script can perform any task that the storage administrator chooses to perform
ranging from sending an e-mail or writing to a log file up to executing a task that adds
additional enabled data devices to the pool in question.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 2
To configure Alerts, pick Alert Settings from the Administration panel. The Alert Settings
screen allows you to view and modify Alert Policies and Alert Thresholds.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 3
To configure Alert Policies, click the Alert Policies big button from the Home >
Administration > Alert Settings page.
The Alert Policies page allows you to configure Symmetrix alerts. Most of the alerts are
disabled by default. All alerts need to be explicitly enabled through this dialog if they are to
be used.
In the Alerts Policies page, highlight the alert categories that you wish to enable and then
click on the Enable button.
One can optionally configure Unisphere to send out notifications when an alert triggers. In
order to do this, click the Notify button. Then, select one or more of the notification types
(e-mail, SNMP or syslog).

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 4
This page shows the alerts that are of relevance to Virtual Provisioning.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 5
There are three categories of alerts that can be triggered by Unisphere for VMAX. They are
warning, critical, or fatal depending on the type of alert and the threshold that is being
equaled or exceeded.
The default policies cannot be altered or deleted.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 6
To configure Alert Thresholds, click the Alert Thresholds big button from the Home >
Administration > Alert Settings page.
The Alert Thresholds page can be used to enable monitoring and setting threshold limits on
the percentage of utilization of TimeFinder/Snap, SRDF/A Delta Set Extension (DSE), and
Virtual Provisioning Thin Pools. When enabling threshold alerts, you can specify a threshold
value (percentage of utilization) for each of the severity levels: Warning, Critical, and Fatal.
For example, if you set warning to 50 for a particular pool, Unisphere will issue a warning
alert when that device pool is at 50% utilization.
The Alert Thresholds page can also be used to configure Unisphere to alert you when an
SRDF/A session has been off loading (spilling) cycle data to a DSE pool for a certain length of
time. The settings are in minutes.
Please note that the default threshold policies cannot be modified. To setup customized
thresholds, click on the Create button. In the Create Thresholds Policies dialog, pick the
Symmetrix system from the dropdown menu, then pick the Category from the dropdown
menu. The categories are DSE/Snap/Thin Pool Utilization and DSE Spill Duration. Then,
highlight the pools to which the policy should apply and choose the threshold levels. Next,
click the OK button to create a customized threshold. Notifications can be optionally setup.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 7
Alerts that have not been cleared by the user can be viewed by clicking the Alerts button.
They can alternately be viewed by clicking the New Alerts button at the upper right of the
browser window. This button will also show the number of new alerts (alerts that have not
been acknowledged by the user).
In the sample shown above, the alerts severity, as displayed by Unisphere for VMAX are:
(1) Fatal
(2) Critical
(3) Warning
(4) Information
(5) Normal

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 8
The event daemon (storevntd) enables the monitoring of Symmetrix operations by detecting
and reporting events as they happen. The event daemon continually collects Symmetrix
event information, filters the events by severity and type, and responds by alerting client
event applications and/or logging events to specified targets.
When using the daemon with a client event application (for example, Unisphere for VMAX),
the application registers with the event daemon, specifying the events in which it is
interested. When used in this manner, the daemon will automatically start when the client
application requests its services.
When configuring the daemon to log events, it is possible to set it up to log the events to a
remote Syslog server, the UNIX Syslog, the Windows Event log, SNMP, and/or a file on disk.
The daemon should typically be configured to automatically start at system boot.
The event daemon is also supported on z/OS as part of the Solutions Enabler installation.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 9
By default, the event daemon will automatically start the first time a Solutions Enabler
application requires its services.
The daemon can be manually started using the stordaemon command:
stordaemon start storevntd
Alternatively, it can be automatically started every time the local host is booted using the
following command:
stordaemon install storevntd autostart

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 10
The event daemon is documented in the Solutions Enabler Installation Guide. The amount of
information is somewhat limited. The daemon_options file has examples of how to set up
the options. The Technical Note has further examples on how to use the event daemon.
Some of the options shown here are undocumented and was taken from Engineering
documentation.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 11
To manually perform monitoring with the event daemon, a user needs to specify a target for
the daemons messages. The targets can be one, or a combination of:
file Events are written to a file on disk.
snmp Events are mapped into SNMP traps.
system Events are written to the local hosts syslog services on Unix. The syslogs
configuration settings determine which log file the message is written to. On Windows the
messages are sent to the Event Log.
syslog Events are sent directly to a remote syslog server, bypassing any local syslog
service.
Special applications, such as Unisphere for VMAX or Control Center, have been written to
take advantage of the event daemon. These applications request the information they need,
and the event daemon supplies it.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 12
The list of events to be logged are specified in the daemon options file. The event daemon
can report on a large number of categories of events which are documented in the Solutions
Enabler Installation Guide. The list of individual event codes are documented in an Appendix
of the Guide.
Certain events correspond to numerical quantities of some sort. A threshold is associated
with each severity level, and an event is generated at that severity when the event's value
exceeds the associated threshold. These fields can be used to override the default threshold
values controlling when an event is delivered.
One example of this is the event that indicates the percentage of space used for a device
pool. These fields can be set to control when events are to be generated, e.g.:
thresh_critical=96, thresh_major=80, thresh_warn=60, thresh_info=40.
An easy way to find the set of supported events and categories is by querying a running event
daemon.
# Load the Symmetrix detector:
stordaemon action storevntd cmd load_plugin Symmetrix
# List the event categories that it supports:
stordaemon action storevntd cmd list -categories

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 13
The event daemon provides the necessary SNMP MIB support and trap generation services
required to monitor the status of Symmetrix storage environments from third-party
enterprise management frameworks. The event daemon includes a loadable SNMP library
which, once enabled and configured in the daemon_options file, acts as a self contained
SNMP agent. It is responsible for maintaining internal Fibre Alliance MIB (V3.0) tables,
responding to SNMP browse requests, and generating traps in response to events.
For an application to receive SNMP trap information from the event daemon, it must be
specified as a trap target. The applications IP address, the port on which the application will
be listening for the trap and the filter that determines the highest severity level for which
traps will be sent.
The possible values range from 1 through 10, where:
1 = Unknown
2 = Emergency
3 = Alert
4 = Critical
5 = Error
6 = Warning
7 = Notify
8 = Info
9 = Debug
10 = Mark (all messages logged)

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 14
The event daemon can be started explicitly by the user or set to start automatically every
time the system boots. When the daemon starts it prints an entry in the event daemon log.
The log file can be dated if the user chooses to enable that option in the daemon options file.
By default there are two log files storevntd.log0 or storevntd.log1. These are
circular logs that grow to 1 MB size before they get overwritten. The event daemon log notes
the options that it read from the daemon options file at the time it started.
A record of events is noted in the events log, which is also a circular file called
events.log0 or events.log1. The file can be given a different name and it can be
dated, if the user chooses to specify these options in the daemon options file.
Sometimes it may be necessary to change the options on a running daemon without
stopping and restarting it. To do this you can edit the daemon options file and issue the
command:
# stordaemon action storevntd cmd reload
Another way is to dynamically set an option in the running daemon is by using the command:
# stordaemon setvar storevntd name var=Value

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 15
The table is excerpted from the Appendix of the installation guide and contains the event
codes related to Thin provisioning. Event codes 1111, 1206, 1207 and 1216 display changes
in state so no additional settings are required. However, codes 1208, 1212 and 1213 require
threshold values that correspond to different severities. The default severities of 60, 65, 70,
80 and 100% apply if user defined severities are not specified.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 16
This is a simpler way of setting up the options in the daemon options file than the
documented way in the Technical Note and in the daemon_options file.
The sev=warning is an undocumented option and specifies that every 1208 (thin pool)
and 1212 and 1213 (thin device) event that generates a warning or something more serious
than a warning should be logged. Other permitted values are:
sev=normal sev=info sev=minor sev=major sev=critical
The storevntd:symm_sync_frequency is an undocumented option. Setting it to 1
causes the event log to be updated every 30 seconds as opposed to every 2.5 minutes. This
option is not really necessary in real life environments, but it is useful for demonstration or
lab environments.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 17
The entries in the daemon options file were read by the event daemon at the time of start
up. The excerpt in the lower half of the page is from the storevntd.log0 or
storevntd.log1 file.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 18
As a result of the options set in the daemon_options file, these entries were recorded in
the event log as two Thin devices and the Thin pool got filled up. Warnings are issued
whenever the thresholds are exceeded. However, you may not see all the warnings at 60%,
65%, 70%, 80% and 100%. If the Thin device or the Thin pool get filled up quicker than the
polling rate of the daemon, a warning or two may be skipped.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 19
This is an example that uses threshold values to specify information and different warning
levels. This is documented in the daemon_options file and the Technical Note.
The options in the daemon options file are read by the event daemon at the time of startup.
While the event daemon is running, these options stay in force.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 20
The two entries in the event log pertain to the enabling and disabling of the Thin data device
pools.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 21
The four levels of messages on this page pertain to the four severities specified in the options
file. The messages are issued once for each Thin device bound to the pool and once for the
pool as a whole.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 22
These are the key points covered in this module. Please take a moment to review them.

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 23
*This slide is intentionally left blank.*

Copyright 2013 EMC Corporation. All rights reserved Module 7: Monitoring Thin Pools 24