IBM Tivoli Monitoring For Network Performance V2.1 The Mainframe Network Management Solution Sg246360

Front cover
IBM Tivoli Monitoring for Network Performance V2.1

The Mainframe Network Management Solution
Managing TCP/IP network performance from z/OS Sample implementation scenarios Operational examples and tips
Budi Darmawan Venugopal Devarasetti Gary Kalatucka Garth Madella Tian Huat Peh Giancarlo Rodolfi
ibm.com/redbooks
International Technical Support Organization

IBM Tivoli Monitoring for Network Performance V2.1: The Mainframe Network Management Solution
October 2004
SG24-6360-00
Note: Before using this information and the product it supports, read the information in Notices on page ix.
First Edition (October 2004) This edition applies to Version 2, Release 1, Modification 0 of IBM Tivoli Monitoring for Network Performance (product number 5698-FNP).
Copyright International Business Machines Corporation 2004. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Chapter 1. Introduction to network performance monitoring . . . . . . . . . . . 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 The automation blueprint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 IBM Tivoli Monitoring for Network Performance . . . . . . . . . . . . . . . . . . . . . 4 1.4 Redbook environment and scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Document organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2. Components and architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Components and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Web application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Web application structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.2 Web application user interface functions . . . . . . . . . . . . . . . . . . . . . 12 2.2.3 Role-based security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.4 Problem determination for Web application . . . . . . . . . . . . . . . . . . . 19 2.3 Monitor functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.1 Process structure of the monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3.2 Files used by the monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.3 Performance data sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.3.4 Setting options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.3.5 Problem determination for the monitor . . . . . . . . . . . . . . . . . . . . . . . 31 2.4 Communication and security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4.1 User authentication mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4.2 Communication port usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.4.3 Certificates and authentication with SSL. . . . . . . . . . . . . . . . . . . . . . 38 2.4.4 Transport between DB2 and monitor . . . . . . . . . . . . . . . . . . . . . . . . 39 2.5 Database structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.5.1 Configuration tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.5.2 Measurement tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Chapter 3. Implementation scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.1 Implementation components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Copyright IBM Corp. 2004. All rights reserved.
iii
3.2 Implementation scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.2.1 Distributed servers environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.2.2 Pure z/OS environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3 Scenario comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.1 Distributed consideration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.3.2 z/OS only consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.3.3 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4 Scenario implementation roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.5 User operation and roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Chapter 4. AIX Web application implementation . . . . . . . . . . . . . . . . . . . . 57 4.1 Web application on AIX overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.2 Preparing for the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2.1 File system space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2.2 Setting up the Java Messaging Service . . . . . . . . . . . . . . . . . . . . . . 61 4.2.3 User authority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.2.4 Enabling DB2 password encryption . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.2.5 WebSphere access to DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3 Web application implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3.1 Implementation steps overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.2 Web application installation details . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.3.3 Setting up LDAP in z/OS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3.4 Configuring WebSphere Application Server for LDAP on z/OS . . . . 74 4.3.5 Verification of the distributed implementation . . . . . . . . . . . . . . . . . . 78 4.4 Some problems and their solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4.1 Problem with console users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.2 LDAP user ID character constraint . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.4.3 Uninstallation from admin console . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.4.4 Set up the X-windows DISPLAY properties to enable graphics . . . . 86 4.4.5 LTPA key generation problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.4.6 Problem with SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.5 Start and stop procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.6 Backup and recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.6.1 File system backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.6.2 DB2 database backup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.6.3 DB2 database restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.6.4 File system restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Chapter 5. Mainframe Web application implementation . . . . . . . . . . . . . 101 5.1 Scenario overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.2 Preparing for the implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.1 Preparing HFS files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.2 Preparing DB2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
iv
5.2.3 Graphic package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.2.4 Preparing WebSphere Application Server . . . . . . . . . . . . . . . . . . . 110 5.3 Web application implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3.1 Installation procedure overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3.2 Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.4 Start and stop procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.4.1 Start up the Web application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.4.2 Shut down the Web application. . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.5 Backup and recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.5.1 Backup and restore of file systems . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.5.2 Backup and restore of DB2 database . . . . . . . . . . . . . . . . . . . . . . . 121 Chapter 6. Monitor implementation and operation . . . . . . . . . . . . . . . . . 125 6.1 Monitor installation process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.1.1 Before the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.1.2 Preparing the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.1.3 Parameters for itmnp.properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.2 Some problems and their solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.2.1 Missing Tivoli common directory . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.2.2 Setting APF authorized attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.3 Start and stop procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.4 Sample monitor configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.5 Monitoring best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 6.5.1 Monitored metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.5.2 Monitoring configuration design . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.5.3 Monitoring information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.5.4 Monitoring storage usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 6.5.5 Monitoring network interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Chapter 7. Discovery and alert interfaces. . . . . . . . . . . . . . . . . . . . . . . . . 161 7.1 NetView Integrated TCP/IP services component . . . . . . . . . . . . . . . . . . 162 7.1.1 Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 7.1.2 Configure NetView Integrated TCP/IP Services Component . . . . . 162 7.1.3 Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 7.2 Event integration with IBM Tivoli Enterprise Console . . . . . . . . . . . . . . . 164 7.2.1 Customizing TEC rulebase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 7.2.2 Configuring event forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 7.2.3 Sample event automation program . . . . . . . . . . . . . . . . . . . . . . . . . 173 7.3 Event integration with IBM Tivoli NetView for z/OS. . . . . . . . . . . . . . . . . 175 7.3.1 Setting up Event Automation Service . . . . . . . . . . . . . . . . . . . . . . . 176 7.3.2 Defining threshold and event generation . . . . . . . . . . . . . . . . . . . . 177 7.3.3 Automating NetView alert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Chapter 8. Historical reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Contents
8.1 Tivoli Data Warehouse overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 8.2 Tivoli Data Warehouse setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 8.2.1 Installation for distributed data source . . . . . . . . . . . . . . . . . . . . . . 186 8.2.2 Installation for mainframe data source . . . . . . . . . . . . . . . . . . . . . . 186 8.3 Installation of the Warehouse Enablement Pack. . . . . . . . . . . . . . . . . . . 187 8.3.1 Back up TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 8.3.2 Warehouse Enablement Pack installation. . . . . . . . . . . . . . . . . . . . 188 8.4 ETL processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 8.4.1 ETL overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 8.4.2 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . 199 8.5 Sample reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 8.5.1 Configuring Crystal Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 8.5.2 Available reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 8.5.3 Accessing the Crystal ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Appendix A. ITMNP configuration files and test programs. . . . . . . . . . . 217 AIX itmnp.properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 z/OS itmnp.properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Sample server TSO REXX program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Sample object REXX client program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Appendix B. z/OS LDAP SDBM back end configuration files . . . . . . . . . 223 z/OS LDAP setup configuration file: ldap.profile . . . . . . . . . . . . . . . . . . . . . . 224 z/OS LDAP configuration file: SLAPDCNF. . . . . . . . . . . . . . . . . . . . . . . . . . . 225 z/OS LDAP started procedure: LDAPSRV . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 Appendix C. Tips collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 WebSphere Application Server Version 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 J2EE based application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Startup and shutdown procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 DB2 Universal Database Version 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Relational database system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 DB2 on z/OS systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 DB2 on AIX or Window systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Differences between z/OS and distributed DB2 processing . . . . . . . . . . . 231 Important tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 z/OS Communication Server TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 IBM Tivoli NetView for z/OS Version 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Alerts structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Automation support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Event automation service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 IBM Tivoli Enterprise Console Version 3.9. . . . . . . . . . . . . . . . . . . . . . . . . . . 235 IBM Tivoli Enterprise Console structure . . . . . . . . . . . . . . . . . . . . . . . . . . 235
vi
IBM Tivoli Data Warehouse Version 1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Data warehouse concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 Warehouse enablement pack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Appendix D. Underlying technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Light-weight Directory Access Protocol (LDAP) . . . . . . . . . . . . . . . . . . . . . . . 242 eXtensible Markup Language (XML) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Microsoft Excel for browsing XML files . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Certificates and encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Secret Key Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Public Key Cryptography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Refinements on cryptographic techniques . . . . . . . . . . . . . . . . . . . . . . . . 256 Secure Sockets Layer protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 The Record Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 The communication protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Contents
vii
viii
Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.
ix
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX Cloudscape CICS Database 2 Domino DB2 Universal Database DB2 Eserver Eserver IBM ibm.com IMS iSeries Lotus MQSeries MVS NetView OS/390 pSeries Redbooks Redbooks (logo) RACF RDN RMF SecureWay System/390 Tivoli Enterprise Tivoli Enterprise Console Tivoli VTAM WebSphere z/OS zSeries
The following terms are trademarks of other companies: Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others.
Preface
This IBM Redbook explains the new IBM Tivoli Monitoring for Network Performance Version 2.1. This version of IBM Tivoli Monitoring for Network Performance provides a complete redesign of the z/OS TCP/IP management tools that was started by the NetView Performance Monitor for TCP/IP. IBM Tivoli Monitoring for Network Performance provides a comprehensive TCP/IP stack monitoring for z/OS. It collects performance metrics from the z/OS Communication Servers system management interface, measuring response time and Simple Network Management Protocol (SNMP) Management Information Base (MIB) variable collection. IBM Tivoli Monitoring for Network Performance uses strategic IBM software platforms, such as WebSphere Application Server, as the Web application platform, and DB2 Universal Database as the central repository. This redbook starts with exploring the architecture of IBM Tivoli Monitoring for Network Performance and its components. We also discuss various implementation scenarios and evaluate the benefits and drawbacks of each scenario. Implementation planning and consideration is presented and operational consideration is explained.
The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center.
xi
Figure 1 The redbook team
Budi Darmawan is a project leader at the International Technical Support Organization, Austin Center. He writes extensively and teaches IBM classes worldwide on all areas of systems management on distributed and z/OS. Before joining the ITSO five years ago, Budi worked in IBM Indonesia as a lead IT Specialist performing implementation services and solution architecting. His current interest is advanced business service management solution. Venugopal Devarasetti is a Software Engineer at IBM Software Labs, Bangalore, India. He has been with IBM for four years, after receiving his Engineering degree from Kuvempu University, India. He has been involved with the IBM Java Development Kit on AIX and z/OS. His area of expertise includes J2EE, Web Services, and JVM Internals. Gary Kalatucka is a Senior IT Consultant for Tivoli Services Americas in the United States. He has 26 years of experience in the z/OS field, including four years of Tivoli software experience. His areas of expertise include z/OS, SNA, z/OS Automation products, and various Tivoli software products. Garth Madella is a Information Technology Specialist with IBM South Africa. He has 18 years of experience in the System/390 networking field. He has worked with IBM for eight years. His areas of expertise include VTAM, SNA TCP/IP,
xii
and sysplex. He has written extensively on TCP/IP and Enterprise Extender issues. Tian Huat Peh is a Advisory IT specialist with IBM Singapore. He has nine years of experience in the OS/390 system and TCP/IP field. His areas of expertise include z/OS USS, OSA-Express implementation, and z/OS TCPIP implementation. Giancarlo Rodolfi is a zSeries Certified Consultant TSS in Brazil. He has 18 years of experience in zSeries field. His areas of expertise include zSeries and Linux. He has written extensively on z/OS Communication Server. Thanks to the following people for their contributions to this project: Wade Wallace International Technical Support Organization, Austin Center Bob Haimowitz and Richard M Conway International Technical Support Organization Laura Knapp and Douglas Gibbs IBM US, Tivoli Software
Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Preface
xiii
Comments welcome
Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at:
ibm.com/redbooks
Send your comments in an Internet note to:

redbook@us.ibm.com
Mail your comments to: IBM Corporation, International Technical Support Organization Dept. 0SJB Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493
xiv
Chapter 1.
Introduction to network performance monitoring

This chapter introduces you to the network performance monitoring, specifically TCP/IP network performance monitoring. It begins by putting performance management in context and develops the discussion of the usage of IBM Tivoli Monitoring for Network Performance in this perspective. The discussion in this chapter is divided into the following sections: 1.1, Introduction on page 2 1.2, The automation blueprint on page 2 1.3, IBM Tivoli Monitoring for Network Performance on page 4 1.4, Redbook environment and scope on page 5 1.5, Document organization on page 5
1.1 Introduction
Mainframes are playing a vital role in the current network computing environment and on-demand initiatives. They serve as the premier servers servicing hundreds and thousands of users. These mainframes are interconnected together and they interact with other machines and services through the network. Primary communication protocol shifted from the traditional Systems Network Architecture (SNA) to the Transmission Control Protocol/Internet Protocol (TCP/IP). This introduces a new challenge for managing the TCP/IP network communication on these mainframe servers. The IBM Tivoli Monitoring for Network Performance provides the facility to manage all aspects of TCP/IP communication from the z/OSs IBM Communication Server TCP/IP. IBM Tivoli Monitoring for Network Performance Version 2.1 brings a new management paradigm on this area. It is a complete re-design of the product from previous versions and NPM/IP product. This redbook discusses the new IBM Tivoli Monitoring for Network Performance Version 2.1 implementation and operation scenarios. Since this is a critical component in the overall system management and automation environment, we will also discuss its role in the automation blueprint.
1.2 The automation blueprint

The IBM Tivoli solution is the base of providing automation on the overall system management for the on demand world. Automation is so critical for businesses to achieve resiliency, efficiency, responsiveness, and flexibility. The IBM automation platform shows the structure of the automation components in providing on demand automation capability. The IBM automation blueprint is shown in Figure 1-1 on page 3.
Business Service Management
Policy Based Orchestration
Availability
Assurance
Optimization
Provisioning
Virtualization
Software Resources System Resources
Figure 1-1 IBM automation blueprint
The IBM automation blueprint is a game-changing plan for reducing the complexity of technology to allow you to focus on the business goals and the application of resources to business objectives rather than the management of technology. The blueprint enables enterprises to implement automation in an evolutionary fashion that acknowledges the heterogeneous nature of the infrastructure. At the bottom of the blueprint is the foundation the Software and System Resources with native automation capabilities required for higher level automation functions. Many of these resources may be virtualized to the other capabilities. Here, the key point is that in order to achieve the highest levels of on demand automation, resources need to be virtualized so that they can be dynamically provisioned as business policies require. Above the resources are the key automation capabilities: Availability helps ensure that systems are available 24x7. Reliance or security keeps your systems protected from threats and provides the functions for a great user experience in accessing applications and data they need while keeping out unwelcome users.
Chapter 1. Introduction to network performance monitoring
Optimization provides tools to make the most of the resources you have so that they are running at peak performance and efficiency and providing you with the maximum return on your investment. Provisioning focuses on the self-configuring, dynamic allocation of individual elements of your IT infrastructure so that Identities or Storage or Servers are provisioned as business needs dictate. The next layer, Policy-based Orchestration, helps customers automatically control all the capabilities of the four areas we just discussed so that the entire IT infrastructure is responding dynamically to changing conditions according to defined business policies. This orchestration builds on the best practices of the customers collective IT experience, and helps to ensure that complex deployments are achieved with speed and quality on demand. Finally, Business-driven Service Management capabilities provide the tools you need to manage service levels, meter system usage and bill customers for that usage, as well as model, integrate, connect, monitor, and manage your business processes end-to-end for complete linkage of IT and business processes. Being able to view IT resources in context of business systems is a unique capability that we need. The IBM Tivoli Monitoring for Network Performance provides availability monitoring for mainframes TCP/IP network servers. It resides in the availability function and provides networking server performance monitoring.
1.3 IBM Tivoli Monitoring for Network Performance

IBM Tivoli Monitoring for Network Performance Version 2.1 provides a centralized monitoring of the TCP/IP networking protocol. IBM Tivoli Monitoring for Network Performance meets your daily tactical needs as well as your long-term strategic systems management goals, providing an effective way to gain control of mission-critical network resources, performance issues, and workload distributions. IBM Tivoli Monitoring for Network Performance provides timely analysis of performance related metrics, such as response time, traffic flow, system workload, and CPU utilization. Using the IBM Tivoli Monitoring for Network Performance Web application, operators can monitor the performance of the network in an effort to anticipate problems and resolve them before they occur. The performance data can be used to detect bottlenecks and other potential problems, which eliminates the need for network systems programmers to manually scan through extensive amounts of performance data. The network systems programmer can use this data to perform detailed problem determination for problems that cannot be
resolved by the network operators. In addition, the network systems programmer can use this data to validate service level agreements and improve network performance through network tuning. After the data is summarized and aggregated into the Tivoli Data Warehouse, the capacity planner can use this data to do trend analysis and forecasting to improve performance and better plan for network growth.
1.4 Redbook environment and scope

This redbook discusses the implementation and operation scenarios of the IBM Tivoli Monitoring for Network Performance Version 2.1. The discussion is divided into two scenarios: the mainframe based scenario and the combination of mainframe and distributed scenario. The detailed environment is presented in 3.4, Scenario implementation roadmap on page 54.
1.5 Document organization

The discussion in this document is divided into the following chapters: Chapter 1, Introduction to network performance monitoring on page 1, this chapter, serves as the book introduction. Chapter 2, Components and architecture on page 7 explains, in detail, the IBM Tivoli Monitoring for Network Performance components and their interaction. Chapter 3, Implementation scenarios on page 47 discusses possible implementation scenarios and how we will discuss them in this redbook. Chapter 4, AIX Web application implementation on page 57 explains the implementation of the AIX-based Web application. Chapter 5, Mainframe Web application implementation on page 101 explains the implementation of the z/OS-based Web application. Chapter 6, Monitor implementation and operation on page 125 explains the monitoring components and some discussion in monitoring process. Chapter 7, Discovery and alert interfaces on page 161 describes integration with IBM Tivoli NetView for IP resource discovery and also alerts usage with IBM Tivoli Enterprise Console or IBM Tivoli NetView for z/OS. Chapter 8, Historical reporting on page 183 explains the data collection implementation for IBM Tivoli Monitoring for Network Performance data.
Chapter 1. Introduction to network performance monitoring
The appendices provide program listing and additional information for your reference in reading this redbook.
Chapter 2.
Components and architecture

This chapter discusses the product architecture of IBM Tivoli Monitoring for Network Performance. This chapter includes the following sections: 2.1, Components and functions on page 8 discusses the primary components of IBM Tivoli Monitoring for Network Performance and their functions. 2.2, Web application on page 10 describes the Web application structure. 2.3, Monitor functions on page 22 explains the monitoring subsystem. 2.4, Communication and security on page 34 shows the communication between the components of IBM Tivoli Monitoring for Network Performance. As it is a distributed application, security of the communication is of a concern. 2.5, Database structure on page 41 describes the underlying data structure for IBM Tivoli Monitoring for Network Performance.
2.1 Components and functions

IBM Tivoli Monitoring for Network Performance monitors the performance of the TCP/IP network in your enterprise from the mainframe perspective. Performance data from all monitored systems are stored in a central DB2 database. The data collected by IBM Tivoli Monitoring for Network Performance can be accessed using the Web application and also be used as source information for historical reports generation using Tivoli Data Warehouse. The basic function of the IBM Tivoli Monitoring for Network Performance is to collect performance data, which requires you to install the IBM Tivoli Monitoring for Network Performance monitor component on each of the z/OS systems that you want to monitor. In addition, you must install the IBM Tivoli Monitoring for Network Performance Web application on a system that has WebSphere Application Server and DB2 database run. The supported operating systems for the IBM Tivoli Monitoring for Network Performance Web application are z/OS and AIX. Figure 2-1 shows the overall IBM Tivoli Monitoring for Network Performance architecture.
NetView Integrated TCP/IP Services Component Windows/AIX
WebSphere Application Server IBM Tivoli Monitoring for Network Performance Web Application DB2
Monitor Configuration
JMS
IBM Tivoli Monitoring for Network Performance monitor z/OS
Tivoli Data Warehouse Server Crystal Enterprise Server
Central Data Warehouse and Data marts
Performance data
IBM Tivoli Enterprise Console IBM Tivoli NetView for z/OS
AIX or z/OS
Windows
Figure 2-1 IBM Tivoli Monitoring for Network Performance components
As shown in Figure 2-1, the IBM Tivoli Monitoring for Network Performance architecture consists of several components. The required components are shown in the darker boxes, while optional components are shown in white boxes and connected by dotted lines.
The required components are: WebSphere Application Server and DB2: The WebSphere Application Server and the DB2 are required for the IBM Tivoli Monitoring for Network Performance Web application. Both components must be run on the same server. WebSphere Application Server provides the platform for running the IBM Tivoli Monitoring for Network Performance Web application. DB2 acts as the central repository that contains a database that holds the configuration and performance data. See 2.5, Database structure on page 41 for more information. Web application: The IBM Tivoli Monitoring for Network Performance Web application runs as a WebSphere Application Server application. It is installed using Install Shield Multi-Platform (ISMP) on AIX or Windows platform or using a script from the UNIX Systems Services command line for z/OS. The product interface is a role-based Web application that is accessible from your Web browser. The Web Application provides the interface to view and configure the IBM Tivoli Monitoring for Network Performance environment. See 2.2, Web application on page 10 for more information. Monitors: The IBM Tivoli Monitoring for Network Performance Monitor is configured and runs on each z/OS operating system. The monitor performs three separate functions: collect performance data for the z/OS operating system, collect SNMP data from configured IP addressable, and collect availability and response time data from IP-addressable resources in the enterprise that the monitor is able to ping. In a normal configuration, there would be multiple instances of the z/OS monitors recording data to the DB2 database for the IBM Tivoli Monitoring for Network Performance Web application. See 2.3, Monitor functions on page 22 for more information. The optional components are: Event receivers: IBM Tivoli Monitoring for Network Performance can be configured to forward events in Event Integration Facility (EIF) format. This means that the events can be forwarded to IBM Tivoli Enterprise Console (TEC) or IBM Tivoli NetView for z/OS. IBM Tivoli Monitoring for Network Performance provides the necessary TEC classes to receive events. Events forwarded to IBM Tivoli NetView for z/OS need to be received using the Event Automation Service. NetView Integrated TCP/IP Services Component (ITSC): It is highly recommended you install the NetView Integrated TCP/IP Services Component, which provides automatic discovery of IP-addressable resources in your enterprise. ITSC can discover and classify any IP-addressable resources that are running SNMP agent. The SNMP information can be queried for additional information, such as resource type, which can be used to group these resources into SmartSets. It can automatically detect z/OS systems and other TCP/IP stack information from z/OS. NetView Integrated
Chapter 2. Components and architecture
TCP/IP Services Component software is provided with IBM Tivoli Monitoring for Network Performance. Tivoli Data Warehouse: IBM Tivoli Monitoring for Network Performance captures performance data that can be collected into Tivoli Data Warehouse. Tivoli Data Warehouse is a strategic cross-application infrastructure for collecting historical data and generating reports. IBM Tivoli Monitoring for Network Performance provides a set of extract, transform, and load (ETL) programs utilities to summarize and migrate historical performance data to Tivoli Data Warehouse. Historical data stored in Tivoli Data Warehouse is used to generate historical performance reports using Crystal Enterprise, which is provided as part of Tivoli Data Warehouse.
2.2 Web application

The Web application is the main component of IBM Tivoli Monitoring for Network Performance. It serves as the central contact point for other components. It uses WebSphere Application Server and DB2 database. This section discusses the structure of the Web application. The discussion is divided into: 2.2.1, Web application structure on page 10 2.2.2, Web application user interface functions on page 12 2.2.3, Role-based security on page 17 2.2.4, Problem determination for Web application on page 19
2.2.1 Web application structure

Figure 2-2 on page 11 shows the overview of the Web application components and their connections.
10
itmnp21 Enterprise Application

JMS messaging JDBC interface
ITMNPDB ITMNP JMX ITMNP EJB
itmnpItsc
SERVLETs
itmnpUI
Web interface
monitors
Netview ITSC nvexportd
Web browser
Figure 2-2 The Web application component structure
The Web application consists of several modules. Some of these modules have external interfaces to interact with other components. The following are the modules of the itmnp21 J2EE application: The itmnpJMX is a Java resource module that contains Java Management eXtension classes. These classes are subclasses that represent the JMX objects that reside in the monitors. This is an internal component of the Web application. This resource package for the JMX connector is stored in a resource archive (rar) file. The itmnpEJB module is an Enterprise Java Beans (EJBs) module for WebSphere Application Server. This module consists of EJBs for various objects that are managed by IBM Tivoli Monitoring for Network Performance, such as nodes, ITSC processes, commands, and so on. These EJBs provide the means to search and retrieve objects from a relational database through Java programs. Some of the objects here communicate using Java Messaging Service (JMS). The itmnpItsc module is the module that uses servlets and interacts with EJBs for IBM Tivoli Monitoring for Network Performance. It provides the interface to the monitors and NetView ITSC. The z/OS version of this module does not support SSL or HTTPS communication. This module utilizes the EJB and JMX modules for IBM Tivoli Monitoring for Network Performance. The itmnpUI module is the Web application user interface module that interacts with an operator using a Web browser. This module contains Java
11
Server Pages and static Web content for the operator Web console. This module can be accessed using either HTTP or HTTPS protocol on all platforms. Database access is performed though the JDBC interface to the DB2 database.
2.2.2 Web application user interface functions

The Web application serves the IBM Tivoli Monitoring for Network Performance user interface to a Web browser. This section describes the user interface functions of the Web application: A product interface for a user to access by browser The user interface is a role-based Web application that is accessible from a Web browser. When security is enabled, a user name and password are required to sign on to the Web application. The initial login window is shown in Figure 2-3. The Administrator Full Access check box allows configuration and maintenance functions to be available when the user is authorized.
Figure 2-3 Initial login window
Function based portfolio When the user is initially logged in, the left side of the window contains the menu portfolio that the user can access. Figure 2-4 on page 13 shows a user
12
with administrator role menu. It has both the display and management menus. The management options are shown in the red box.
Figure 2-4 Portfolio menu for an administrator
Maintains and manages the configuration for the monitors The Web application provides three configuration wizards to create the monitor definition. The wizards are for: z/OS monitor SNMP monitor Availability and response time monitor The following are the steps that you will go through with the wizard: Defines one or more monitor locations The monitor location table contains the z/OS system name and IP address (or fully qualified host name) of one or more monitors in your enterprise.
13
The IP addresses or host names that you enter into this table are used by the IBM Tivoli Monitoring for Network Performance Web application when communicating with the monitor component. Provides a list of resources to monitor The list of resources to monitor will differ dependent on the type of monitor definition you are creating. z/OS monitor definition: You will specify the list of TCP/IP stacks on each z/OS system that you want to monitor. By default, you will monitor the TCP/IP stack you provided when you defined the monitor locations. If the z/OS systems you are monitoring has multiple TCP/IP stacks, you will provide the IP address or fully qualified host name of each additional stack that you want to monitor. The TCP/IP stacks must reside on the same z/OS system where the monitor resides. SNMP monitor definition: You will specify the IP address or fully qualified host name of each resource you want to monitor. You must have network connectivity to each of these resources from the monitor and the SNMP agent must be running on each of these resources in order to retrieve data using an SNMP query. Availability and Response Time monitor definition: You will specify the IP address or fully qualified host name of each resource you want to monitor for availability and response time. In order to collect availability and response time data, you must be able to ping each of these resources from the monitor. Availability and Response time monitor definitions should be created to monitor the most critical IP resources in your environment.
Specifies the type of data to collect The type of data to collect will differ depending on which type of monitor definition you are creating. z/OS monitor definition: You will choose from a list of 10 different categories of data to collect, such as TCP, IP, UDP, FTP, and so on. SNMP monitor definition: You will chose from a list of MIBS that contain performance data. Availability and Response Time monitor definition: Availability and response time data will be collected for each of the resources you are monitoring.
Specifies threshold and rearm values Threshold values can be specified for each monitored metric. The threshold value is used to determine the point to which an alert is generated. Each piece of performance data that is collected by the monitor is compared to the threshold value to determine if the threshold
14
has been crossed for all data that has been displayed on the user interface. If the threshold has been crossed, a red indicator will be placed next to the value of that metric when it is displayed. In addition, you can choose to generate an event when a threshold is crossed. Define schedule: Collection time and interval A schedule entry defines the day of the week and time that you want to start collecting data and the day of the week and time that you want to stop collecting data. Additionally, for each schedule entry, you will define a collection interval and frequency. Note: Data collection for FTP and TN3270 sessions happens immediately following the completion of the session. This data is not stored based on the interval. Multiple monitor definitions that request the same data are combined to collect the data only once. When different collection definitions contain different schedules for similar data, IBM Tivoli Monitoring for Network Performance data collection engine must choose which intervals are used. The basic algorithm for collecting z/OS Communications Server data is as follows: i. It selects the shortest collection interval from all the collection instruction. ii. If more than one definition is currently active that have the same collection interval, it chooses one based on the order in which the definitions were delivered to the monitor component, meaning the order is an arbitrary choice by the monitor. Sets operation preferences for the environment You must set the operational values for your environment before creating and deploying the monitor configurations. Doing so ensures that the monitor collects performance data and generates events. The following are the operation preferences: Database purge preferences The Database Purge Preferences task is used to define the number of days to retain data in the database, the time of day the daily purge occurs, and whether the purge is dependent upon completion of the ETL process. SNMP preferences The SNMP preferences task must be defined in order to collect performance data from SNMP resources in your enterprise. The SNMP agent must be running on each IP resource that you want to monitor.
15
Define event receivers. An event is generated when specific thresholds are crossed. Events can be viewed using IBM Tivoli Enterprise Console or any applications that is capable of receiving and displaying events. The event receiver defines the fully qualified host name or IP address and port for the IBM Tivoli Enterprise Console Server or the IBM Tivoli NetView for z/OS. Data view in graphics and table format The IBM Tivoli Monitoring for Network Performance Web application shows performance data in table and graphical format view of the performance data collected in the DB2 database. Figure 2-5 shows the table view of the performance data. This is the default view mode.
Figure 2-5 Table view of performance data
Figure 2-6 on page 17 shows the graphic view of the performance data. For more information on getting the graph, refer to the IBM Tivoli Monitoring for Network Performance Operator Guide, SC31-6365.
16
Figure 2-6 Graphical view
Note: The graphic engine for a z/OS Web application server uses a shareware program called pja toolkit, while for an AIX based Web application server it uses X-Windows graphic classes; therefore, the DISPLAY environment variable must be set to refer to a server with an active X-server.
2.2.3 Role-based security

The user interface is secured with a set of roles. The Web application roles are configured using the WebSphere Application Server Web-based administration.
17
An authenticated user ID and password are required to sign in to the Web application. Depending on the platform and authentication mechanism, the creation of the user IDs are the responsibility of the security administrator. The user IDs are then associated with a role in the Web application. The IBM Tivoli Monitoring for Network Performance Web application uses three roles for user assignment: ITMNP Operators can view collected performance metrics for various configured monitors through the Web application. ITMNP Admin can configure and manage monitors to collect performance data, and set run-time preferences and global defaults for the product. ItscAllAuthority is used for connecting to the itmnpItsc module; this is for the monitor or NetView ITSC component when it connects to the Web application. The default is the ItscAllAuthority is assigned to everyone. Do not change this, as the monitors may fail. User authentication is handled by WebSphere Application Server. This can be a local operating system user ID or a remote LDAP directory server user ID. Most installations may want to use a z/OS based user ID to be consistent. The user ID needs to be assigned to a role in WebSphere Application Server. Role assignment is performed using WebSphere Application Servers administrative console. Figure 2-7 on page 19 shows the role assignment dialog for IBM Tivoli Monitoring for Network Performance.
18
Figure 2-7 Role assignment for the Web application
2.2.4 Problem determination for Web application

All components of the Web application run on a single Java Virtual Machine (JVM) started by the WebSphere Application Server. This JVM has the standard output and standard error redirected to files called SystemOut.log and SystemErr.log respectively. Those files reside under the path: <WebSphere directory>/AppServer/logs/<server name> where: WebSphere directory The path where WebSphere is installed; on AIX, it is typically /usr/WebSphere. server name The WebSphere server name; on AIX, it is typically called server1. Our z/OS server is called ws611sc61.
19
You can activate tracing for a WebSphere component to get more detail on the processing of the component from the WebSphere Application Server administrative console. From the administrative console, select Troubleshooting Logs and Trace, as shown in Figure 2-8.
Figure 2-8 Trace menu
Select the server name that you want to modify and select the Diagnostic Trace property. The diagnostic trace setting dialog is shown in Figure 2-9 on page 21.
20
Figure 2-9 WebSphere trace setting
In Figure 2-9, the trace is enabled for Java Messaging Services and it is writing to a file with a size of 20 MB. You may need to modify this size, as it may not be enough for a busy system. If you want to modify the components to be traced, click on the Modify button and you get the setting page, similar to Figure 2-10 on page 22.
21
Figure 2-10 Setting component trace
Click on the Apply, Close, OK, Save, and then Save again to save the setting. For more information, refer to IBM Tivoli Monitoring for Network Performance: Messages and Troubleshooting, SC31-6366.
2.3 Monitor functions

The IBM Tivoli Monitoring for Network Performance monitor component runs on z/OS systems to collect TCP/IP network performance data and send the collected data to a DB2 database. The monitor performs the following functions: Collects performance data from the z/OS system where the monitor resides. The z/OS performance data is stored in a central DB2 database and can be displayed using the IBM Tivoli Monitoring for Network Performance Web application.
22
Collects SNMP performance data from IP-addressable resources in the network that are running the Simple Network Management Protocol (SNMP) agent. The SNMP data cannot be shown using the Tivoli Monitoring for Network Performance Web application, but it is stored in the DB2 database. Collects availability and response time data from IP-addressable resources in the network. It uses the ping command or ICMP protocol to collect this information. Availability and response time data is stored in the central database and can be displayed using the IBM Tivoli Monitoring for Network Performance Web application. Sends events when performance metrics indicate a possible problem in the network. The events are sent in Event Integration Facility (EIF) format. The monitor configuration defines which resources to monitor, which types of data to collect, when to start and stop collecting data, and how often to collect data. A monitor configuration can consist of one or more monitor definition. The monitor configuration is defined using the Web application.
2.3.1 Process structure of the monitor

The monitor process structure is shown in Figure 2-11.
itm np M o n ito r.sh
L a un che r
M o n ito r_ cs3 9 0
Java V irtu al M ach in e
IT M N P M o nitor (U S S )
z/O S C o m m un ica tion S e rve r
S N M P d ae m on
z/O S
W e b A p plicatio n
D B2
Figure 2-11 Monitor startup process
23
As shown in Figure 2-11 on page 23, the processes for the monitor are: A shell script, itmnpMonitor.sh, initializes environment variables before starting the launcher. The launcher starts the two main processes running in z/OS Unix System Services (USS), and a C-language program monitor_cs390 collects performance data from z/OS Communication Server and a Java Virtual Machine (JVM) that communicates with the Web application and DB2. The JVM establishes the communication link with the Web application through the itmnpItsc servlets and with DB2 through Java Database Connectivity (JDBC). The JVM reads the monitor configuration and setup data collection. The JVM process also collect SNMP and ICMP monitors. The monitor_cs390 collects data for the z/OS Communication server API. It retrieves data for TCP/IP stack, FTP, TN3270, TCP applications, TCP connections, TCP storages, UDP endpoints, Enterprise Extender (EE) and High Performance Routing (HPR), Common Storage Manager (CSM) storage, and so on.
2.3.2 Files used by the monitor

The files used by the IBM Tivoli Monitoring for Network Performance monitor process is shown in Figure 2-12.
$tivoli_common_dir/FNP/ logs/fnp_config.xml
$DBCacheDirectory/dbcache
$CONFIG_DIR/Itmnp.properties
ITMNP Monitor
$tivoli_common_dir/FNP/logs/msg_fnp_monitor*.log
$FNPlogPropertiesLocation /etc/ibm/tivoli/common/cfg/log.properties
$tivoli_common_dir/FNP/logs/trace_fnp_monitor*.log
Figure 2-12 The monitor file usage
24
From Figure 2-12 on page 24, the monitor uses the following configuration information: itmnp.properties This is the main configuration file. The location of this file is under the $CONFIG_DIR path, as specified in the startup script for the monitor. This configuration file contains the path to the Tivoli common directory. It typically resides in the /etc/ibm/tivoli/common/cfg; however, if you set the environment variable $FNPlogPropertiesLocation in the startup script, you can specify the location of the file. This is the configuration of the monitoring process. This file is retrieved from the Web application when the monitor is started. This configuration is used to build the objects for metric collection. It resides in the common log directory.
log.properties
fnp_config.xml
The monitor uses the following files for output: dbcache This is the location of information caching in a Cloudscape database before being written to DB2. The location of this directory is specified in the variable $DBCacheDirectory from the itmnp.properties
msg_fnp_monitor*.log and trace_fnp_monitor*.log These are the messages and traces log files, which reside under the tivoli common directory, as specified in the log.properties file. The configuration XML file, fnp_config.xml, is the latest configuration XML file that the monitor has received from the Web application. It is stored in the same directory as the log files. The structure of the XML file is shown in Figure 2-13 on page 26.
25
Poller Configuration Document Poller Configuration Collection CS390 Collection CS390 Target CS390 Data UDPTable FTP TN3270 TCPDetail TCPStor TCPListen Threshold Value CS390Attributes CS390 System Data CSMStor EEHPT CS390 System Target Interval Event Forwarding Collection SNMP Collection SNMP Expression MIBData
SNMP Expression MIBData
Interval Event Forwarding
Figure 2-13 Structure of fnp_config.xml
As shown in Figure 2-13, the configuration of the monitor is split into multiple collection instructions. Each collection represents the requirement for data and the target table monitor to collect. There are three types of collection: OS390 collection, which collects data from the z/OS communication server SNMP collection, which acquires MIB data from SNMP ICMP collection, which records availability and response time data Each collection is further qualified with the interval definition and event forwarding destination information.
2.3.3 Performance data sources

The monitor uses a set of network management interfaces provided by the z/OS Communications Server to collect most of the z/OS performance data. The remaining performance data is collected using SNMP queries for performance data that is stored in MIBs. SNMP data can be collected from any IP-addressable resource in the enterprise that is running the SNMP agent. Tivoli Monitoring for Network Performance has provided a specific set of performance metrics that you can choose from when creating an SNMP monitor definition. For a complete list of the SNMP performance metrics supported by IBM Tivoli Monitoring for Network Performance, refer to Appendix B, SNMP Performance Data, in the IBM Tivoli Monitoring for Network Performance Version 2 Release 1 Administrator Guide, SC31-6364.
26
Note: Some of the z/OS performance data is collected using SNMP queries. To ensure that you collect the necessary performance data, you must have the OSNMPD agent running on each of the z/OS TCP/IP stacks that you want to monitor. Availability and response time data can be collected from any IP-addressable resource in the enterprise that the monitor is able to ping. The monitor uses the ping command to determine the availability of the resource. It uses the ping response times to calculate an average response time for each resource being monitored. The data collection schedule is determined by the start and end times and the intervals specified in the monitor definitions. The interval determines how often the monitor will collect data. For example, if the interval is set to 30 minutes, the monitor will collect data every 30 minutes. This is true for all types of data except for FTP data and data that is displayed on the TN3270 Server Sessions screen. FTP and TN3270 Server Session data is provided by the z/OS Communications Server as events occur. As a result, the data on these screens will not change according to the specified interval. Table 2-1 shows where the performance data is collected from.
Table 2-1 Performance data sources Performance data category TCP stack Data source SNMP V2 Management Information Base for Transmission Control Protocol using SMIv2 and z/OS Communications Server management interfaces. SNMPv2 Management Information Base for the User Datagram Protocol using SMIv2. SNMPv2 Management Information Base for the Internet Protocol using SMIv2. z/OS Communication Servers management interfaces. z/OS Communication Servers management interfaces. Interface group MIB. IBM OSA-Express Direct SNMP Enterprise-Specific MIB.
UDP stack IP stack TN3270 and FTP sessions HPR and EE information Interface Adapter interface
27
Performance data category Memory data Response time SNMP performance data
Data source z/OS Communications Server management interfaces. Result of ping command. Use SNMP queries to collect performance data that is stored in MIBs.
More discussion on monitors and the data related to them can be found in 6.5.1, Monitored metrics on page 138.
2.3.4 Setting options

The itmnp.properties file contains configuration parameters for the monitor. This file is typically copied to /etc/itmnp/ directory. This directory is specified in the variable $CONFIG_DIR within itmnpMonitor.sh shell script. The itmnp.properties file consists of the configuration parameters for the monitor. The detailed information for each parameters are provided as comments in the member; see our examples in AIX itmnp.properties on page 218 and z/OS itmnp.properties on page 218. The parameters can be grouped into: Monitor configuration monitor_hostname The fully qualified domain name of the system where the monitor is located. This name is used to retrieve the monitor configuration from the Web application and must match the monitor name provided when defining the monitor configuration. The interface or stack which the monitor binds on this host. The first of the two ports used by the monitor for internal communication. The monitor uses the port and the next numbered port (such as 1670 and 1671). This is the port monitor_cs390 process is listening on.
bind_interface CSAPIport
Communication between the monitor and Web application; see also 2.4, Communication and security on page 34 local_httpport The port number that the monitor listens on, and the Web application used to communicate with the monitor. This port is used by the Web application to transmit the monitor configuration.
28
WAS_hostname WAS_httpport
The host name of the WebSphere Application Server where the Web application runs. The port number that the monitor uses to communicate with the Web application.
socksServer, socksPort The fully qualified URL and port for the Socks server the monitor will use to communicate with the WebSphere Application Server. (Optional) Database properties DBName DBUserName DBPassword DBHostName The name of the DB2 database where the performance data will be stored. The database user who has the authority to store data in the DB2 database. The password of the DB2 database user. The fully qualified host name of the system where DB2 is located. This is the same system as the WebSphere Application Server host name. The port number that DB2 database server is listening on. Typical value for AIX is 50000, while in z/OS you can see it from the DSNL004I message in the DB2 startup. The type of driver used to establish connections with the database. Use the value of UNIVERSAL.
DBPort
DBDriverType
Tip: Although this parameter defaults to DBDriverType=UNIVERSAL, we found that we need to comment it out for a DB2 for z/OS database. Explicitly specifying DBDriverType=UNIVERSAL make the connection to a distributed DB2 database. DBCacheDirectory The directory to be used by the monitor to store the collected data in a local cache. This cache stores the collected data before it is sent to the DB2 database. This directory is only used when the DBCachRowLimit and DBCacheTimeout are > 1. The maximum number of seconds that the monitor will store data in a local cache before sending the data to the DB2 database. Data will be transferred to the DB2 database when either the maximum number of seconds is reached or the maximum number of rows is reached.
DBCacheTimeout
29
DBCacheRowLimit
The maximum number of rows of data that the monitor stores in a local disk cache before attempting to store the data in the DB2 database. Use Cloudscape as a cache. This may improve the performance for high volumes systems. This is the setting for the security level between the monitor and DB2 database. See 2.4.4, Transport between DB2 and monitor on page 39.
enableCloudscape db2Security
SSL properties; all the keyStore and trustStore settings are for HTTPS protocol with the Web application resides on AIX. WebSphereServletProtocol The protocol that the monitor uses to communicate with the Web application. For non-secure, it is HTTP, and for secure, it is HTTPS. trustStoreName The name of the file where the WebSphere Application Server and monitor certificates are stored. The name of the file where the WebSphere Application Server and monitor keys are stored. The password to access the key store file.
trustStorePassword The password used to access the trust store file. keyStoreName keyStorePassword
keyManagerPassword The password to access the key manager interface. This should be the same as keyStorePassword. Message and trace log properties; see 2.3.5, Problem determination for the monitor on page 31 for more details trace.maxFiles The maximum number of trace files that may exist at one time. When the monitor closes a full trace file, it will ensure that no more than the maximum number exists by deleting the oldest trace file before creating a new trace file. When the number is 1, the file size limit is ignored. You can specify individual type (Java, CLI, or C) separately. The maximum size (in kilobytes) each trace file may contain before the monitor closes the old trace log and creates a new trace log. You can specify individual type (Java, CLI, or C) separately. The maximum number of message files that may exist at one time. When the monitor closes a full message file, it will ensure that no more than the maximum
trace.maxFileSize
message.maxFiles
30
number exists by deleting the oldest message file before creating a new message file. You can specify individual type (Java, CLI, or C) separately. message.maxFileSize The maximum size (in kilobytes) each message file may contain before the monitor closes the old trace log and creates a new message log. You can specify individual type (Java, CLI, or C) separately. monitor.trace.level The logging level for the trace logs. This controls how much trace data is collected. The possible values are DEBUG_MIN, DEBUG_MID, and DEBUG_MAX.
2.3.5 Problem determination for the monitor

For more information on problem determination, refer to IBM Tivoli Monitoring for Network Performance: Messages and Troubleshooting, SC31-6366. This section contains our experience on performing the troubleshooting. In debugging the monitors, the main information sources are the XML trace and log files. These files are located under the Tivoli common directory, typically /var/ibm/tivoli/common/FNP/logs. This directory contains the following files: fnp_config.xml: The current configuration file; if this file does not exist, the monitor initialization may fail or the monitor cannot communicate with the Web application. msg_fnp_monitor*.log: The message file for the JVM process shown in Figure 2-11 on page 23. The database connection and configuration file processing messages are here. The number of files and their size are governed by the parameters message.Java.maxFiles and message.Java.maxFileSize. trace_fnp_monitor*.log: Trace file for detailed information on the JVM process; the number of these files and their size are governed by the parameters trace.Java.maxFiles and trace.Java.maxFileSize. msg_fnp_monitorc*.log: The message file for the monitor_cs390 and the launcher process, as shown in Figure 2-11 on page 23. The calls to Communication Server API messages are shown here. The number of files and their size are governed by the parameters message.C.maxFiles and message.C.maxFileSize. trace_fnp_monitorc*.log: Trace file for detailed information on the monitor_cs390 process; the number of this files and their size are governed by the parameters trace.C.maxFiles and trace.C.maxFileSize.
31
Note: The parameters trace.maxFiles, trace.maxFileSize, message.maxFiles, and message.maxFileSize contain the default value for the derivative trace.*.maxFiles, trace.*.maxFileSize, message.*.maxFiles, and message.*.maxFileSize parameters. The tracing levels provided are DEBUG_MIN, DEBUG _MID, and DEBUG_MAX. We recommend running the processes with the DEBUG_MIN level, as these files have an XML format and are huge. In our environment, a two minute interval produces more than 50 MB of trace files with a DEBUG_MAX level. The serviceability consideration must be take into account when allocating the HFS dataset. The trace level can also be modified using a Command Line Interface. You start the CLI using the command itmnpMonitor.sh cli. This will give the menu shown in Figure 2-14.
VBUDI @ SC66:/itmnp/V2R1M0/bin>./itmnpMonitor.sh cli Launching ITMNP Monitor CLI Main Menu Current state: Running Configuration last loaded: Tue Jun 15 12:59:00 EDT 2004 Build level: Thu Apr 29 09:00:00 2004 1) Change trace logging levels 2) Suspend collection of all measurements 3) Resume collection of all measurements 4) Exit Type your selection number: Figure 2-14 Monitor CLI menu
The trace level is changed by selecting 1 from the menu in Figure 2-14. This gives the screen in Figure 2-15 on page 33.
32
Change Trace Logger Levels Menu 1) All loggers 2) CS390 Data Layer DEBUG_MIN 3) CS390 Layer DEBUG_MIN 4) CS390 Socket Data Layer DEBUG_MIN 5) Command Line Interface DEBUG_MIN 6) Database Layer DEBUG_MIN 7) ICMP Layer DEBUG_MIN 8) JMX Layer DEBUG_MIN 9) Monitor Base Code DEBUG_MIN 10) Monitor CS390 - Base DEBUG_MIN 11) Monitor CS390 - CSM Storage DEBUG_MIN 12) Monitor CS390 - EE and HPR DEBUG_MIN 13) Monitor CS390 - FTP and TN3270 DEBUG_MIN 14) Monitor CS390 - Hash table DEBUG_MIN 15) Monitor CS390 - SNA Management DEBUG_MIN 16) Monitor CS390 - TCP Applications DEBUG_MIN 17) Monitor CS390 - TCP Connections DEBUG_MIN 18) Monitor CS390 - TCP Storage DEBUG_MIN 19) Monitor CS390 - UDP Endpoints DEBUG_MIN 20) Monitor Services Code DEBUG_MIN 21) Monitor utilities DEBUG_MIN 22) Monitor utilities - Socket DEBUG_MIN 23) SNMP Layer DEBUG_MIN 24) Return to previous menu. Type your selection number: Figure 2-15 Trace setting
As an example, if you want to change the CS390 Data Layer setting, select 2 and then specify the trace level, as shown in Figure 2-16.
Type your selection number: 2 Select New Logging Level For: CS390 Data Layer 1) DEBUG_MIN 2) DEBUG_MID 3) DEBUG_MAX 4) Return to previous menu. Type your selection number: Figure 2-16 Trace level
33
2.4 Communication and security

As discussed in 2.1, Components and functions on page 8, IBM Tivoli Monitoring for Network Performance is a distributed application. With such a structure, communication is a vital part. Components of IBM Tivoli Monitoring for Network Performance communicate using the TCP/IP protocol, which raises some concern about the security of the management infrastructure. The platform selection for the Web application, on AIX or a z/OS system, determines the available options for securing the communication. The discussion in this section is divided into: 2.4.1, User authentication mechanism on page 34 2.4.2, Communication port usage on page 35 2.4.3, Certificates and authentication with SSL on page 38 2.4.4, Transport between DB2 and monitor on page 39
2.4.1 User authentication mechanism

The IBM Tivoli Monitoring for Network Performance Web application relies on WebSphere Application Server for the user authentication. It supports two security authentication mechanisms: For AIX, IBM Tivoli Monitoring for Network Performance supports authentication using Lightweight Directory Access Protocol (LDAP) or local operation system (LocalOS). For z/OS, IBM Tivoli Monitoring for Network Performance supports authentication using LocalOS through z/OS Security Server or Resource Access Control Facility (RACF) or another System Authorization Facility (SAF)-compliant product. The WebSphere Application Server security authentication mechanism supports implementation for most major LDAP directory servers or LDAP servers that can be used as the repository for user and group information. These LDAP servers are called by the WebSphere Application Server for authenticating user and other security related entities. If you are running IBM Tivoli Monitoring for Network Performance in a z/OS environment, you can use RACF to perform native authentication using z/OS V1 R2 SecureWay Lightweight Directory Access Protocol (LDAP) Server. This arrangement makes it easy for the various components of IBM Tivoli Monitoring for Network Performance to access the information without having to replicate the data in other repositories.
34
The z/OS implementation of this protocol provides the ability to use user and group information and update passwords in the z/OS Security Server or RACF. There are two directory access mechanisms provided for the RACF LDAP server: Traditional Database Manager (TDBM) or Security Database Manager (SDBM). This means that when an application attempts to authenticate an entry in the LDAP server, the password verification is performed by the SDBM information in RACF. If the password that was sent to the LDAP server matches the password that is stored in the Security Server user ID associated with that entry, then the application is authenticated as the DN of the directory entry in TDBM. Figure 2-17 shows the communication between SDBM and TDBM requests for a LDAP client.
Authentication Request Via Web application
LDAP Client
Bind()
LDAP Server
SDBM TDBM _ACLs
_passwd()
RACF
DB2
Figure 2-17 A z/OS LDAP server using an SDBM/TDBM configuration
The basic usage of z/OS native authentication of IBM Tivoli Monitoring for Network Performance Web application does not requires a TDBM database configuration.
2.4.2 Communication port usage

Transport communication using TCP/IP can be prone to eavesdropping and spoofing. In this section, we discuss various port usage that can be mapped to IBM Tivoli Monitoring for Network Performance components depending on the encryption mechanism. Understanding communication port assignment is important to putting a firewall between the components. By default, the transport between the monitor and WebSphere Application Server is non-secure, meaning that the communication transport between the monitors
35
and WebSphere Server is not encrypted and not authenticated. If WebSphere Application Server is running on AIX, you can secure the transport by encrypting communication to and from the Web application.
Non-secure communication
In a non-secure configuration, the Hyper Text Transfer Protocol (HTTP) is used. Figure 2-18 describes the communication flows and the ports used.
9080
IBM Tivoli Monitoring for Network Performance Web Application
J2C
9081
IBM Tivoli Monitoring for Network Performance Monitor
Netview Integrated TCP/IP services component
z/OS
WebSphere Application Server
HTTP DB2
38030 for z/OS 50000 for AIX
z/OS or AIX
Figure 2-18 Non-secure communication ports configuration
In Figure 2-18, the following communication configuration happens: The monitor contacts WebSphere Application Server for configuration information on port 9080 using non-secure HTTP protocol. Port 9080 is specified with the was_httpport parameter in the itmnp.properties configuration file. The Java Messaging Extension (J2C) inside WebSphere Application Server responds on non-secure port 9081 using the HTTP protocol. Port 9081 is specified with the local_httpport parameter in the itmnp.properties configuration file. The NetView Integrated TCP/IP Services Component (ITSC) sends information IP resources to the Web application on non-secure port 9080 using HTTP. Port 9080 is specified with the WebsphereServletPort parameter in the nvexportd.properties file. Users accessing the Web application uses non-secure (HTTP) port 9080.
36
Secure communication
In a secure communication configuration, the HTTP secure (HTTPS) is used. HTTPS is a protocol based on Secure Socket Layer (SSL). Figure 2-19 shows the default ports and communication flows for a secure configuration for AIX Web application. Note: For the z/OS Web application, the communication between the WebSphere Application Server and the monitor cannot be HTTPS. IBM Tivoli Monitoring for Network Performance does not support authentication and encryption of this communication transport when running WebSphere Application Server and the DB2 product on a z/OS system. The Web application console user will still have an HTTPS session through port 9443.
9454 HTTPS
IBM Tivoli Monitoring for Network Performance Web Application WebSphere Application Server
J2C
9455 HTTPS 9454 HTTPS
Netview TCP/IP services component

9453 HTTPS
z/OS
DB2
50000
AIX
Figure 2-19 Secure communication ports configuration for AIX Web application
In this configuration: The monitor contacts WebSphere Application Server for configuration information on port 9454 using HTTPS. Port 9454 is specified with the was_httpport parameter in the itmnp.properties configuration file. The Java Messaging Extension (J2C) inside WebSphere Application Server responds on port 9455 using HTTPS. Port 9455 is specified with the local_httpport parameter in the itmnp.properties configuration file. Additional information that must be provided in the itmnp.properties file includes the TrustStore file name and password and the KeyStore file name and password.
37
The NetView Integrated TCP/IP Services Component sends information IP resources to the Web application on secure port 9454. Port 9454 is specified with the WebsphereServletPort parameter in the nvexportd.properties file. Additional information that must be provided in the nvexportd.properties file includes the TrustStore name and password and the KeyStore name and password.
Internal port used by monitor

The monitor component requires two ports for internal communications. The monitor will use the port specified on the CSAPIport parameter in the monitor itmnp.properties file as the first port. The second port used for internal communication will be the next numbered port. The default value for CSAPIport is 1670, which means ports 1670 and 1671 will be used by the monitor for internal communications. These ports are used regardless of a non-secure or secure configuration.
2.4.3 Certificates and authentication with SSL

The IBM Tivoli Monitoring for Network Performance includes a set of self-signed client and server certificates. These certificates are provided in .jks format as well as .arm format. The files itmnpMonitorCert.arm and itmnpServerCert.arm contain exported self-signed certificates that the security administrator can import into existing certificate stores and key rings by using tools such as IBM Global Security Kit (GSKIT) or the keytool command supplied with the Java Development Kit (JDK). Prior understanding of Secure Sockets Layer (SSL) and certificates is assumed; for more information on SSL and certificates, refer to Appendix D, Underlying technology on page 241. Table 2-2 shows the SSL trust stores, key stores, and certificates required to create secure SSL connections. You can also use this table to replace the self-signed certificates with certificates from a trusted Certificate Authority (CA).
Table 2-2 SSL related files and certificates for secure connection Repositories WebSphere Application Server Monitor TrustStore itmnpServerTrustStore.jks Trusted Certificate itmnpServerCert.arm itmnpItsccert.arm itmnpMonitorCert.arm itmnpServerCert.arm KeyStore itmnpServerKeyStore.jks
itmnpMonitorTrustStore.jks
itmnpMonitorKeyStore.jks
38
Repositories NetView Integrated TCP/IP Services Component
TrustStore itmnpItscTrustStore.jks
Trusted Certificate itmnpServerCert.arm
KeyStore itmnpItscKeyStore.jks
The server certificates are installed when you install the Web application on the WebSphere Application Server. The server key store file contains the public and private keys for the server. The server trust store file contains the self-signed certificates for the monitors and NetView Integrated TCP/IP Services Component. The client certificates are installed when you install the monitor component. The client key store file contains the public and private key of the client. The client trust store file contains the monitors and servers self-signed certificates. The secure ports used by the Web application and the monitor are described in 2.4.2, Communication port usage on page 35. To secure the transport between WebSphere Application Server and the monitors, you must specify values for the trustStoreName and keyStoreName parameters in the itmnp.properties file for each monitor. To secure the transport between the NetView Integrated TCP/IP Services Component and WebSphere Application Server, you must copy the client certificates from the IBM Tivoli Monitoring for Network Performance CD-ROM and store them on the server where the NetView Integrated TCP/IP Services Component is running. You must then specify values for the trustStoreName and keyStoreName statements in the nvexportd.properties file. If you choose to secure the transport, all of the monitors and systems where the NetView Integrated TCP/IP Services Component is running must be configured to secure the transport.
2.4.4 Transport between DB2 and monitor

The DB2 product uses one port to listen for communication from the monitors. The port number is specified using the DBPort parameter in the monitor itmnp.properties file. The default value for this port is 50000 for AIX. Note: The discussion in this section applies to a Web application running on AIX.
39
By default, IBM Tivoli Monitoring for Network Performance uses password-protected security when it transmits data to and from the DB2 database. A user ID and password are verified at the DB2 server, and the password is transmitted using CLEAR_TEXT_PASSWORD_SECURITY. Security may be increased by configuring for encrypted passwords. Security may be reduced by configuring for CLIENT verification. Communication between the monitors and the DB2 database can be secured using security mechanisms provided by the DB2 server. One of these mechanisms should be used to protect against unauthorized access to your databases. The DB2 product supports five security mechanisms, including two that require Kerberos. However, the IBM Tivoli Monitoring for Network Performance does not support Kerberos as an option. As a result, only three of the mechanisms are supported in this environment. The following DB2 security mechanisms are supported by IBM Tivoli Monitoring for Network Performance. The security parameter db2Security in itmnp.properties determines how and where authentication of a user takes place. The supported types are: CLIENT: Authentication takes place at the client. As a result, authentication is not performed at the server. This is the IBM-supplied default value for IBM Tivoli Monitoring for Network Performance. SERVER: User ID and password are sent from the client to the server so that authentication can take place on the server SERVER_ENCRYPT: User ID and password are sent from the client to the server so that authentication can take place on the server. This is the same as SERVER, except that any passwords sent over the network are encrypted. The IBM Tivoli Monitoring for Network Performance monitor component includes a DB2 universal driver that allows the monitors to communicate with the DB2 server from a remote system. The DB2 universal driver configuration options reside in the itmnp.properties file, known as DBDriverType. The names of these security mechanisms do not exactly match the names of the security mechanisms on the DB2 server. However, they are related. Table 2-3 on page 41 contains the supported DB2 server mechanisms and the universal driver security mechanisms that are compatible with each one.
40
Table 2-3 DB2 universal driver configuration options DB2 Server Security Mechanism CLIENT SERVER SERVER_ENCRYPT Values for db2Security statement in the itmnp.properties file USER_ONLY_SECURITY CLEAR_TEXT_PASSWORD_SECURITY CLEAR_TEXT_PASSWORD_SECURITY ENCRYPT_PASSWORD_SECURITY ENCRYPT_USER_AND_PASSWORD_SECURITY
2.5 Database structure

IBM Tivoli Monitoring for Network Performance relies heavily on the data stored in the DB2 database central repository. We feel that it is helpful to understand some of the data structure in the database. This section serves as an overview of the database structure. Refer to the actual installed database for a detailed database structure. The database itself can be considered to consist of two distinct set of tables: The configuration tables, which consists of configuration information from the Web application The measurement tables, which contain the metrics and thresholds for network performance
2.5.1 Configuration tables

The tables in this area define most of the collection and node information. The relationship between tables can be seen in Figure 2-20 on page 42.
41
SMARTSET_OBJ SS_INT_RELN
NETWORK_OBJ
SS_NODE_RELN SS_OBJ_RELN NODE_OBJ
T_SET_SS_RELN INTERFACE_OBJ NETVIEW_ID_MAP
T_SET_NODE_RELN
T_SET_IFACE_RELN
TARGET_SET T_SET_IP_RANGE T_SET_IP_WILD T_SET_HN_WILD EVENT_DEST PRIM_POLL_PARM COL_EV_DEST_RELN MONITOR_INST_CFG
COL_SET_RELN INTERVALGRP INTRVL_TBL
SNM_POLL_PARMS
COLL_TBL COL_INTERVAL_RELN COL_ATTR_RELN COL_REARM_RELN
MONITOR_COL_RELN
COL_THRESH_RELN
COL_ATTR_GRP
REARM_GRP
THRESH_GRP
ICMP_COL_ATTR
SNMP_REARM_EXPR
CS390_REARM_EXPR
ICMP_THRESH_EXPR
SNMP_COL_ATTR
CS390_COL_ATTR
ICMP_REARM_EXPR
SNMP_THRESH_EXPR
CS390_THRESH_EXPR
BEST_PRACTICES
MIB_OBJ
MIB_EXPR
Figure 2-20 Configuration tables database structure
In Figure 2-20, the lines with names are relationships that were stored as tables, while the dotted lines are relationships by content of a key column. Some of the interesting tables are: NODE_OBJ INTERFACE_OBJ SMARTSET_OBJ All TCP/IP addressable nodes that are either entered manually or discovered using NetView ITSC. Interface list of all discovered nodes. Smart-set list of smart set objects in NetView ITSC.
42
NETVIEW_ID_MAP NETWORK_OBJ TARGET_SET COLL_TBL
List of NetView engines that discover the configuration. Network list that is discovered by the NetView ITSC engine. A collection of nodes and interfaces that can be monitored in a single collection. A collection list, which is the primary configuration table for collection. This relates to the collection tag in the fnp_config.xml (see Figure 2-13 on page 26). List of event destinations that are configured from the Web application run-time preferences.
EVENT_DEST
MONITOR_INST_CFG List of monitor instances and active collection in them. INTERVL_TBL List of schedules, that together with the INTERVALGRP table, build the monitoring schedule. This makes up the Interval tag in the fnp_config.xml file. A link to either SNMP_COL_ATTR, ICMP_COL_ATTR, or CS390_COL_ATTR, depending on the collection type. As shown in Figure 2-13 on page 26, a collection can be a CS390 collection, ICMP collection, or SNMP collection.
COLL_ATTR_GRP
REARM_GRP, THRESH_GRP A link to the threshold and rearm tables for CS390, ICMP, and SNMP collection. BEST_PRACTICES MIB_OBJ, MIB_EXPR Tables for MIB specification. There are additional tables that are not mentioned in the diagram; they are: LAST_ETL_RUN ENUM_TYPES This is used by Tivoli Data Warehouse as a processing bookmark. Enumeration value list. Link from SNMP collection to MIB objects to be collected.
2.5.2 Measurement tables

The measurement tables are a set of tables with the suffix of _MSMT. Most of these tables have matching tables with the suffix of _THSH, which contains information for measurements that cross the threshold. The following are in the measurement table list: CSM_SUMM_MSMT CSM storage summary CSM_MON_MSMT CSM storage monitoring
43
FTP_SESS_MSMT
FTP sessions
FTP_CTRANS_MSMTFTP Client transfer records FTP_STRANS_MSMTFTP Server transfer records HPR_AVL_MSMT HPR_PIPE_MSMT HPR_TT_MSMT EE_AVL_MSMT EE_TT_MSMT ICMP_RTT_MSMT IF_STATUS_MSMT IF_MULTI_MSMT HPR availability and response time HPR pipe measurement HPR throughput and traffic EE availability and response time EE throughput and traffic ICMP response time Interface status Multicast/broadcast performance metrics
EE_TT_DET_MSMT EE throughput and traffic detailed measurement
IF_UNICAST_MSMT Unicast performance metrics OSA_ETH_TT_MSMTOSA Ethernet throughput and traffic OSA_STATUS_MSMTOSA status OSA_TT_MSMT OSA throughput and traffic OSA_TT_UTIL_MSMTOSA utilization SNMP_EXPR_MSMT SNMP expression collection SNMP_INT_MSMT STK_IP_TT_MSMT SNMP integer MIB collection IP Stack Throughput and Traffic SNMP_STRING_MSMTSNMP string MIB collection STK_TCP_AVL_MSMTTCP stack availability and response time STK_TCP_TT_MSMT TCP stack throughput and traffic TCP_APP_AVL_MSMTApplication availability and response time TCP_CONN_AVL_MSMTConnection availability and response time TCP_CONN_TT_MSMTConnection throughput and traffic TCP_PRV_MEM_MSMTPrivate memory TN3270_AVL_MSMT TN3270 session availability TN3270_RESP_MSMTTN3270 sliding-window response time TN_RTT_BKT_MSMT TN3270 response time counts by time bucket TN_RTT_BND_MSMTTN3270 response time counts by bound session STK_UDP_TT_MSMT UDP stack throughput and traffic
44
UDP_EP_TT_MSMT UDP endpoint throughput and traffic TRACERT_MSMT Trace route performance
Measurement tables that do not have a corresponding threshold tables are TN_RTT_BND_MSMT and OSA_TT_UTIL_MSMT.
45
46
Chapter 3.
Implementation scenarios
This chapter discusses the implementation scenarios for IBM Tivoli Monitoring for Network Performance. We first evaluate the different pieces that made up IBM Tivoli Monitoring for Network Performance and come up with the scenarios and discuss each of them. The discussion is divided into: 3.1, Implementation components on page 48 3.2, Implementation scenarios on page 48 3.3, Scenario comparison on page 50
47
3.1 Implementation components

The implementation of IBM Tivoli Monitoring for Network Performance can be thought of as a series of implementations of its components. There are several components that are of interest, as they provide options on how they can be implemented. They are: Web application platform The Web application can run on a z/OS or AIX server. This has several implications, as there are differences in the architecture of the operating systems and the application itself. Implementation on AIX will introduce a new operating system on the management platform. Event receiver The event receiver can be IBM Tivoli Enterprise Console or IBM Tivoli NetView for z/OS, depending on which platform the existing automation resides; the decision must be made to optimize the existing infrastructure. Historical data Currently, IBM Tivoli Monitoring for Network Performance only supports Tivoli Data Warehouse to collect and report historical data. If your current historical reporting utilizes Tivoli Decision Support for OS/390, you need to create your own collection programs. Communication security The highly distributed nature of IBM Tivoli Monitoring for Network Performance makes the communication critical. Some components of IBM Tivoli Monitoring for Network Performance support encryption using SSL and HTTPS, while some do not.
3.2 Implementation scenarios

Based on the component implementation selection in 3.1, Implementation components on page 48, we choose two scenarios to implement in this project. The scenarios are: Distributed servers managing a z/OS environment z/OS based implementation The discussion here is divided into the following sections: 3.2.1, Distributed servers environment on page 49 3.2.2, Pure z/OS environment on page 50
48
3.2.1 Distributed servers environment

In this environment, we bring in several non z/OS servers that will be used to manage our z/OS environment. The implementation scope is shown in Figure 3-1.
Netview Integrated TCP/IP Services Component Win2K - Laredo
Websphere Application Server IBM Tivoli Monitoring for Network Performance Web Application
WAS JMX
DB2ITMNPDB
Monitor Configuration Performance data
z/OS SC62
Tivoli Enterprise Console Server Win2K - Laredo
AIX - Jakarta
Win2K - Phoenix
Figure 3-1 Components for IBM Tivoli Monitoring for Network Performance
The configuration in Figure 3-1 consists of: IBM Tivoli Monitoring for Network Performance monitors running on z/OS machines. The IBM Tivoli Monitoring for Network Performance Web application component with DB2 and WebSphere Application Server are installed on an AIX machine to monitor z/OS machines. The IBM Tivoli Enterprise Console server receives events when specific thresholds are crossed in IBM Tivoli Monitoring for Network Performance. NetView Integrated TCP/IP Services Component (ITSC) provides support to automatically discover IP-addressable resources in the enterprise. The IP-addressable resources can be very helpful when configuring the IBM Tivoli Monitoring for Network Performance monitors in your enterprise. A Tivoli Data Warehouse control server with Crystal Enterprise running, which collects data captured by IBM Tivoli Monitoring for Network Performance. The data can be summarized and migrated into the Tivoli Data Warehouse. The historical data is distributed to the user using Crystal Enterprise through a Web browser.
Chapter 3. Implementation scenarios
49
Communication between these components supports either HTTP or HTTPS.
3.2.2 Pure z/OS environment

In this environment, we use a z/OS system to act as the management server. We run the Web application and IBM Tivoli NetView for z/OS there. We still need a non-z/OS platform for this solution for the following items: The historical reporting portion of IBM Tivoli Monitoring for Network Performance for Tivoli Data Warehouse Network object discovery using NetView ITSC Our configuration is shown in Figure 3-2.
z/OS WTSC61 Tivoli Data Ware house DB2 ITMNPDB DB2 TWH_CDW Netview Integrated TCP/IP Services Component z/OS WTSC52
Crystal Enterprise product
MQ (JMS) WMQX
Itmnp21 Web Appication
itmnpMonitor
WebSphere
EAS
Netview
itmnpMonitor
Figure 3-2 z/OS configuration
3.3 Scenario comparison

The comparison in this section is divided into the following sections: 3.3.1, Distributed consideration on page 51 3.3.2, z/OS only consideration on page 52
50
3.3.3, Summary on page 53
3.3.1 Distributed consideration

The IBM Tivoli Monitoring for Network Performance has the ability to run in a distributed way. In this scenario, we have the IBM Tivoli Monitoring for Network Performance monitor component running on zSeries, and the Web application component running on pSeries. This distributed approach can be really useful: When you have pSeries machines being used in the enterprise and have enough capacity to install the Web Application component on it. Refer to IBM Tivoli Monitoring for Network Performance: Planning, Installation, and Configuration, SC31-6363 for the AIX system requirements. Having WebSphere and DB2 skills is not a necessity, but in some enterprises, those skills does not belong to the zSeries personnel. If the enterprise already has the WebSphere Application Server and DB2 running on a pSeries machine running AIX, you could install the Web Application component (WebSphere application Server and DB2) on it to save time. You can use the same WebSphere Application Server and DB2; do not forget that the WebSphere Application Server and DB2 must be in the same operating environment. You can save some zSeries CPU cycles if you deploy the Web Application component on a pSeries machine. The cost of software acquisition. WebSphere Application Server and DB2 for AIX are included with IBM Tivoli Monitoring for Network Performance. When you have requirements for a security configuration, with authentication and SSL communication between the monitor and the browser with the Web Application, you have to run the Web Application on a pSeries machine under AIX. On the other hand, running IBM Tivoli Monitoring for Network Performance in a distributed way can bring more complexity to the scenario. There are some considerations you have to look at: Managing a distributed environment will require more skills, perhaps more people, and more than one team. Do not forget that other components like Tivoli Data Warehouse can be used on the solution and this can include other platforms like Windows 2000. Most enterprises have different teams responsible for each one of the platforms. Troubleshooting in a distributed environment can be really difficult. As you have the components running on different operating systems and, sometimes, remote locations, you will have to deal with network problems
51
and different types of operating systems, each one with its own problems and peculiarities. If you are using a LDAP server to authenticate the users and this LDAP server is not available for any reason, you will not be able to log on to the Web Application to look at the data being collected. Losing one of the components could really cause some problems for the whole solution. There is no rule to define if a distributed scenario is the best way to implement a ITMNP solution. You have to consider a lot of enterprise factors to decide which implementation will better fit your enterprise.
3.3.2 z/OS only consideration

As shown in 3.2.2, Pure z/OS environment on page 50, all the major components of IBM Tivoli Monitoring for Network Performance can run on zSeries: the Monitor component and the Web Application component. Running a mainframe only scenario can have some advantages: You have only a single platform to manage. All the major components will be running on a single environment, which means you have to work with only one platform. This will require less people being involved with the solution. Of course, there are at least three major components or subsystems to be considered here: z/OS Communication Server, WebSphere Application Server, and DB2. The backup and recovery implementations on the mainframe platform tend to be more reliable, and most of the enterprises have them fully implemented, especially if they have DB2 and WebSphere Application Server already running. From the troubleshooting perspective, as everything is running on the same environment, it is easier to do problem determination. From the network perspective, if there are remote systems on the configuration, the network could be a concern. The new machines z890 and z990 have a new CP that can play an important role while running Java applications on a z/OS 1.6 using JDK 1.4 (the zAAP processor). If you have those prerequisites and are planning to implement Java applications on a zSeries machine, this can be really helpful, saving some CPU cycles from the CPs and not paying additional software costs for the additional CPU capacity The drawback of running a mainframe only scenario is: If the enterprise is not eligible to run the zAAP processor, you will have some additional CPU consumption to run this application, and maybe some
52
additional software costs. The distributed scenario can be favorable in this manner.
3.3.3 Summary
In this section, we try to summarize the comparison between the benefits and drawbacks of each solution. The results are shown in Table 3-1.
Table 3-1 Solution comparison Area Software licenses z/OS only WebSphere Application Server and DB2 need to be acquired separately. The main components reside in a single platform, although they run on UNIX System Services. Additional skill is required for Tivoli Data Warehouse administration. The core function needs some z/OS system programmer skill for WebSphere, DB2, and network. Distributed server WebSphere Application Server and DB2 for AIX are already included with the product. Needs to administer monitors running on UNIX System Services; Web application server running on AIX; Event management on IBM Tivoli Enterprise Console and Tivoli Data Warehouse for historical reporting. The core function needs a team of administrators for these different functions: z/OS network administrator, Tivoli Framework administrator and AIX system administrator. Offloading the Web application function in AIX may save some memory and CPU constraint on a zSeries partition, which typically has a higher price compared to a pSeries machine.
Administration skill
Hardware consideration
Having the Web application running on z/OS machine requires a Java environment. JVM in general requires a large amount of virtual memory and the semi interpreted nature of Java requires more CPU cycles. This may result in a dedicated logical partition (LPAR) for the management system, unless the system has a zAAP processor that helps the Java based application run more efficiently. Secured HTTPS communication only supported with the Web console user. The communication with the monitor and NetView ITSC is not secure (HTTP).
Communication requirement
All communication to and from the Web application can be secured using the HTTPS protocol.
53
Area Authentication
z/OS only Native local OS authentication.
Distributed server Distributed authentication using LDAP to the z/OS security server or a local AIX authentication. Backup needs to be performed on the monitors (z/OS) and Web application with its database (distributed).
Backup and recovery
Backup process can be coordinated purely from z/OS perspective.
3.4 Scenario implementation roadmap

We decided to make the scenario similar to the overall configuration shown in Figure 3-3.
Mainframe scenario
Distributed scenario
ITMNP monitor
ITMNP monitor
ITMNP monitor
ITMNP monitor
z/OS SC52
z/OS SC62
z/OS SC66
ITMNP Web application WebSphere Application Server + JMS
ITMNP Web application WebSphere Application Server + JMS
IBM Tivoli Enterprise Console IBM Tivoli NetView ITSC
DB2
DB2
Windows 2000 - laredo AIX - jakarta
Tivoli Data Warehouse Crystal Enterprise Windows 2000 - kingston
IBM Tivoli NetView for z/OS z/OS SC61
Tivoli Data Warehouse Crystal Enterprise Windows 2000 - phoenix
Figure 3-3 Overall implementation environment
The implementation discussion is divided into the following: Distributed Web application implementation is discussed in Chapter 4, AIX Web application implementation on page 57. This includes the following: Preparation for the Web application implementation
54
Web application installation tips Backup procedure for the Web application Mainframe z/OS implementation is discussed in Chapter 5, Mainframe Web application implementation on page 101. This includes the following: Preparation for the Web application implementation Web application installation tips Backup procedure for the Web application Monitor implementation is discussed in Chapter 6, Monitor implementation and operation on page 125. This includes the following: Preparation for the monitor implementation Monitor implementation tips Monitor configuration discussion Backup procedure for the monitor Discovery and alert interface implementation is discussed in Chapter 7, Discovery and alert interfaces on page 161. This includes the following: IBM Tivoli NetView for discovery IBM Tivoli Enterprise Console as the event receiver IBM Tivoli NetView for z/OS as the event receiver Historical reporting with Tivoli Data Warehouse is discussed in Chapter 8, Historical reporting on page 183. This includes the following: Installation process overview for distributed systems Installation process overview for z/OS database Warehouse Enablement Pack installation Running the ETL program Sample reports
3.5 User operation and roles

IBM Tivoli Monitoring for Network Performance can be used by various users and, depending on their roles, they may use different parts of the overall solution. Basically, the users of IBM Tivoli Monitoring for Network Performance can be categorized into: Network system programmer The network system programmer uses IBM Tivoli Monitoring for Network Performance as both an online
55
monitoring tool and historical analysis of performance and capacity information. They need access to the Web application for configuring the monitor, monitoring the result, and managing the historical data. Network administrator The distributed network administrator will be interested in getting information on how the network performs in regard to accessing the mainframe applications and resources. Network operator The network operator can monitor network availability through the graphical interface for IBM Tivoli Monitoring for Network Performance Web application and get information about potential performance problems. The capacity planner, looking at the collected historical data in Tivoli Data Warehouse, can predict the network communication usage and capacity for years to come.
Capacity planner
Service Level Coordinator The service level coordinator uses the information collected from IBM Tivoli Monitoring for Network Performance to ensure no service level violation can and will occur. Database administrator The database administrator needs to help the IBM Tivoli Monitoring for Network Performance maintainer to manage the database. Automation specialist The automation specialist evaluates events from IBM Tivoli Monitoring for Network Performance and programs specific responses for automating problem resolution. Web site administrator The Web master uses performance information collected by IBM Tivoli Monitoring for Network Performance to improve the Web pages that he/she administers. Security administrator The security administrator coordinates the security of the access to the IBM Tivoli Monitoring for Network Performance application, specifically on the z/OS platform.
56
Chapter 4.
AIX Web application implementation

This chapter documents our installation experiences on implementing IBM Tivoli Monitoring for Network Performance on AIX in a distributed environment scenario. The following topics are covered in this chapter: 4.1, Web application on AIX overview on page 58 4.2, Preparing for the installation on page 59 4.3, Web application implementation on page 66 4.4, Some problems and their solutions on page 82 4.5, Start and stop procedures on page 90 4.6, Backup and recovery on page 91
57
4.1 Web application on AIX overview

The distributed scenario discussed in 3.2.1, Distributed servers environment on page 49 uses an AIX Web application server. This chapter deals with the implementation of this AIX-based IBM Tivoli Monitoring for Network Performance Web application. The Web application in this scenario manages the monitors running on SC62 and SC66. The overall structure of this scenario and the host name associated with it are shown in Figure 4-1. The DB2 and WebSphere Application Server are installed on AIX machine jakarta. The implementation is using a secure transport between the monitors and the Web application.
Netview Integrated TCP/IP Services Component Win2K - Laredo
Websphere Application Server IBM Tivoli Monitoring for Network Performance Web Application
WAS JMX
DB2ITMNPDB
Monitor Configuration Performance data
z/OS SC62
Tivoli Enterprise Console Server Win2K - Laredo
AIX - Jakarta
Win2K - Phoenix
Figure 4-1 Distributed environment for IBM Tivoli Monitoring for Network Performance
As shown in Figure 4-1, there are some additional machines that we use. We have a Windows 2000 system, laredo, with NetView Integrated TCPIP Services Components (ITSC) and IBM Tivoli Enterprise Console (TEC) installed on it. Tivoli Enterprise Console Server The Tivoli Monitoring for Network Performance monitors can be configured to generate events to notify operators when specific thresholds are crossed. Events can be received by the IBM Tivoli Enterprise Console server. To use TEC as a receiver, you must create an IBM Tivoli Enterprise Console RuleBase with IBM Tivoli Monitoring for Network Performance classes.
58
NetView Integrated TCP/IP Services Component NetView Integrated TCP/IP Services Component provides support to automatically discover IP-addressable resources in the enterprise. The IP-addressable resources can be very helpful when configuring the IBM Tivoli Monitoring for Network Performance monitors in your enterprise. We have a Windows 2000 system, phoenix, with Crystal Enterprise Server and Tivoli Data Warehouse. The performance data captured by IBM Tivoli Monitoring for Network Performance can be summarized and migrated into the Tivoli Data Warehouse, the Tivoli strategic cross-application infrastructure for archiving data and generating reports. The IBM Tivoli Monitoring for Network Performance uses a set of extract, transform, and load (ETL) utilities to summarize and migrate historical performance data. The ETL files are SQL scripts that execute from the Tivoli Data Warehouse server. ETLs summarize and migrate data into the Tivoli Data Warehouse central data warehouse database and then onto the Tivoli Monitoring for Network Performance data mart database. Data from the data mart can be used to generate historical reports with the Crystal Enterprise product, which is provided as part of Tivoli Data Warehouse. The report files are read by the Crystal Enterprise Server to produce reports. Reports can be viewed from any Web browser through a connection to the Crystal Enterprise Server.
4.2 Preparing for the installation

Before we install the IBM Tivoli Monitoring for Network Performance Web application, we need to preinstall WebSphere Application Server V5 and DB2 Universal Database Version 8 on the AIX machine. We went through the software and hardware prerequisites in Appendix A, Software and hardware prerequisites, in IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363 and made sure that we met the prerequisites, especially the required APARs.
4.2.1 File system space

The installation of the Web application uses an Install-Shield Multi Platform (ISMP) wizard. The wizard can install the database and the Web application automatically. You need to understand where the files are created and prepare sufficient file system space in the AIX machine.
Chapter 4. AIX Web application implementation
59
Checking allocated file systems

In AIX, the physical disk is partitioned in file systems, typically the Journaled File Systems (JFS). For z/OS users, this is much like the Hierarchical File Systems (HFS) or ZFS in UNIX System Services. Use the df commands to get the allocated and free sizes of the file systems, as shown in Example 4-1. The -k option gives the result in kilobytes instead of 512 byte segments.
Example 4-1 Checking file systems in AIX # df -k Filesystem 1024-blocks /dev/hd4 16384 /dev/hd2 983040 /dev/hd9var 163840 /dev/hd3 1425408 /dev/hd1 16384 /proc /dev/hd10opt 32768 /dev/lv00 2457600 /dev/lv01 5242880 /dev/lv02 458752 /dev/lv03 65536 /dev/lv04 16384 /dev/lv05 1310720 /dev/lv06 802816 /dev/lv07 49152 /dev/lv08 425984 /dev/lv09 114688 /dev/lv10 131072 Free %Used 0 100% 12036 99% 143364 13% 358640 75% 15816 4% 24228 27% 151580 94% 4449812 16% 13828 97% 54396 17% 15824 4% 95044 93% 306624 62% 28024 43% 372992 13% 110516 4% 64712 51% Iused %Iused Mounted on 1726 22% / 29402 12% /usr 702 2% /var 80 1% /tmp 26 1% /home - /proc 356 5% /opt 6208 1% /image 740 1% /home/db2inst1 8560 8% /usr/opt/db2_08_01 79 1% /home/dasusr1 18 1% /home/db2fenc1 398 1% /usr/sys/inst.images 16266 9% /usr/WebSphere/AppServer 1087 9% /usr/IBMHttpServer 19 1% /itmnp 29 1% /usr/ibm/tivoli 281 1% /opt/IBM/ITMNP21
File system needed for installation

Table 4-1 shows the file systems that will be used by the installation process. This includes the file systems for the prerequisite products.
Table 4-1 File systems used by the Web application installation Path name /usr/WebSphere/AppServer /opt/IBM/ITMNP21 /tmp /home/db2inst1 Free space 117 MB 53 MB 115 MB 120 MB Our size 350 MB 53 MB 200 MB 512 MB Remark WebSphere path for the Web application IBM Tivoli Monitoring for Network Performance supplied files Temporary installation files IBM Tivoli Monitoring for Network Performance database files
60
The path names in Table 4-1 on page 60 may be included in other file systems in higher level names. We recommend that you create these file systems in advance before installing WebSphere Application Server and DB2. We include the total size of our file systems as a guide for pre-allocating them. For detailed information about planning database space, see Appendix G, Database Sizings and Configuration for Tivoli Monitoring for Network Performance, in IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363.
Changing file system allocation

In AIX, the procedure for modifying the file systems (and also most of the system administration tasks) are performed using the System Management Interface Tool or smit command. This tool can be launched in a text based interface or an X-Windows based Graphical User Interface. For file system administration, use the command smit jfs. This will bring up the JFS menu, as shown in Figure 4-2.
Journaled File Systems Move cursor to desired item and press Enter. Add a Journaled File System Add a Journaled File System on a Previously Defined Logical Volume Change / Show Characteristics of a Journaled File System Remove a Journaled File System Defragment a Journaled File System
Figure 4-2 Smit JFS menu
You can create or modify a file system through this menu. Remember that the size that you specify is always in 512 byte (0.5 KB) units.
4.2.2 Setting up the Java Messaging Service

The Java Messaging Service (JMS) is a stripped down version of WebSphere MQ for message exchange. This feature of WebSphere Application Server must be installed on the AIX machine. To be able to install this feature, you need to prepare the following: 1. Create the mqm and mqbrkrs groups. Run the command smit group to do this. The Group management window is shown in Figure 4-3 on page 62.
61
Groups Move cursor to desired item and press Enter. List All Groups Add a Group Change / Show Characteristics of a Group Remove a Group
Figure 4-3 Group menu
2. Choose Add a Group and press Enter. The Add Group dialog is shown in Figure 4-4.
Add a Group Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] [mqm] false [] [root] [root]
* Group NAME ADMINISTRATIVE group? Group ID USER list ADMINISTRATOR list
+ # + +
Figure 4-4 Add Group dialog
3. Specify the group name of mqm and press Enter. 4. Perform the same thing for group mqbrkrs. 5. Press Esc+0 to exit the smit display. If you have an operational WebSphere Application Server, you should verify that JMS is operational by looking at the SystemOut.log, which is located in <WAS_HOME>/logs/server1. In our case, the SystemOut.log is in /usr/WebSphere/AppServer/logs/server1. Examples of the JMS messages are shown in Example 4-2, JMS messages in SystemOut.log on page 63.
62
Example 4-2 JMS messages in SystemOut.log [5/12/04 [5/12/04 business [5/12/04 [5/12/04 12:28:51:475 CDT] JMSEmbeddedPr MSGS0050I: Starting the Queue Manager 12:28:59:851 CDT] JMSEmbeddedPr MSGS0051I: Queue Manager open for 12:28:59:871 CDT] JMSEmbeddedPr MSGS0052I: Starting the Broker 12:29:04:802 CDT] JMSEmbeddedPr MSGS0053I: Broker open for business
To make sure that the WebSphere Application Server is up and running, find the following message in SystemOut.log:
Server server1 open for e-business
4.2.3 User authority

The user ID that performs the installation needs to be authorized to the file systems and the DB2 database. Authorization to the WebSphere Application Server is acquired using a user ID and password that is specified in the wizard. Typically, when WebSphere Application Server is installed, security is disabled so that there is no authentication needed. We perform the installation using the root user. The root user needs access to DB2 database, which is owned by the so-called DB2 instance owner. Typically, the instance owner user is called db2inst1 with group db2grp1. Unless you connect root to the DB2 owner group, the database will not be created by the wizard. Perform the following to add root to the DB2 group db2grp1. Note: This should be performed after DB2 has already been installed. 1. In AIX session, log in as root. 2. Run the command smit user, which gives the screen shown in Figure 4-5 on page 64.
63
Users Move cursor to desired item and press Enter. Add a User Change a User's Password Change / Show Characteristics of a User Lock / Unlock a User's Account Reset User's Failed Login Count Remove a User List All Users
Figure 4-5 The smit user display
3. Choose Change / Show Characteristics of a User, which gives you the screen shown in Figure 4-6.
Change / Show Characteristics of a User Type or select a value for the entry field. Press Enter AFTER making all desired changes. [Entry Fields] [root]
* User NAME
Figure 4-6 Change / Show characteristics of a User
4. Put root in the User NAME field and press Enter. This gives you the screen shown in Figure 4-7 on page 65.
64
Change / Show Characteristics of a User Type or select values in entry fields. Press Enter AFTER making all desired changes. [TOP] * User NAME User ID # ADMINISTRATIVE USER? Primary GROUP Group SET ADMINISTRATIVE GROUPS ROLES Another user can SU TO USER? SU GROUPS HOME directory Initial PROGRAM User INFORMATION EXPIRATION date (MMDDhhmmyy) Is this user ACCOUNT LOCKED? [MORE...36] [Entry Fields] root [0] true [system] <dit,lp,db2grp1,mqm,mq> [mqbrkrs,mqm] [] true [ALL] [/] [/usr/bin/ksh] [] [0] false + + + + + + +
Figure 4-7 Root user properties
5. Add db2grp1 to the Group SET and press Enter to apply the changes (shown highlighted in Figure 4-7). When the command status is OK, press Esc+0 to exit from the smit interface.
4.2.4 Enabling DB2 password encryption

DB2 password encryption level is an instance wide setting. It can be changed using the command line by the DB2 instance owner. The command is:
db2iupdt -a authType instanceName
where: authType instanceName Can be CLIENT, SERVER, or SERVER_ENCRYPT The instance owner ID, typically db2inst1
We do not use this facility.
65
4.2.5 WebSphere access to DB2

The WebSphere Application Server setting needs to be modified to allow access to the DB2 systems. Note: The data source modification to enable encryption needs to be performed after the Web application is already installed. 1. Include the DB2 JDBC driver: a. From the WebSphere Administrative Console left-hand navigation, select Environment Manage WebSphere Variables and choose the variable DB2_JDBC_DRIVER_PATH. b. Verify or set the path to the file db2java.zip file; the file is typically found in /home/db2inst1/sqllib/java12. c. Click OK and Save. Then click Save again to save this value to the Master Configuration. 2. Add the DB2 password encryption: a. From the WebSphere Administrative Console left-hand navigation, select Resources JDBC Providers. b. Select the DB2 JDBC Provider link and then select the DataSources property. c. Click the itmnpds datasource and choose Custom Properties. d. Click on New to create a new property called securityMechanism. The value is an integer which is: 7 9 For encrypted password For encrypted user and password
e. Click OK and Save. Then click Save again to save this value to the Master Configuration.
4.3 Web application implementation

This documents our installation experiences for implementing IBM Tivoli Monitoring for Network Performance Web application on AIX. The discussions are divided into: 4.3.1, Implementation steps overview on page 67 4.3.2, Web application installation details on page 68
66
4.3.1 Implementation steps overview

The installation of the Web application is performed by an Install-Shield Multi Platform (ISMP) wizard. The wizard can install the database and the Web application automatically. The detailed installation options and selection is provided in Chapter 5, Preparing for and installing the Web application on an AIX system, of IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363. The database creation wizard performs the following functions to create our database, which is called ITMNPDB: 1. Creates the database and connects to it. 2. Creates bufferpools for the ITMNPDB database from itmnpBufferPools.sql. 3. Creates tablespaces for the ITMNPDB database from itmnpTableSpaces.sql. 4. Runs itm_np_ddl to build the configuration tables in the ITMNPDB database. 5. Runs measurements.sql to build the measurement tables in the ITMNPDB database. 6. Creates triggers using itmnp_triggers.sql in the ITMNPDB database. 7. Imports static metadata from preload.sql into the ITMNPDB database. 8. When the database creation is successful, it displays the following message:
FNPI0200I: The IBM Tivoli Monitoring for Network Performance database has been created
The Web application installation wizard performs the following functions: 1. Sets up two message queues. A message queue connection factory, a destination, and a listener port are defined for the first message queue (for com.tivoli.ItscNotify). A message queue connection factory and a destination are defined for the second message queue (for com.tivoli.ItmnpRealtimeResponse). 2. Sets up a JDBC datasource called itmnpds, which refers to the ITMNPDB created by the database wizard. 3. Installs itmnp21.ear file as a new Enterprise Application. 4. Configures WebSphere Java Management eXtension (JMX) connector. Once the above steps are performed, the WebSphere Application Server is recycled to make the Web application available. The Web application will not be available unless the WebSphere Application Server is recycled. The Web application installation wizard for AIX also loads a set of files into the main directory /opt/IBM/ITMNP21 and the sub-directories. This directory is only
67
used for maintenance and it is not use for the actual Web application processing. The sub-directories are: _jvm This directory contains the Java Runtime library (jre) and its licenses. _uninst This directory contains the uninstalled program (uninstall_itmnp21aix.bin) and the data to uninstall the Web application, after you have successfully installed the Web application in AIX. bin This directory contains sample programs, the Web application installation log, the digital certificates for the monitor and Web application server, and DB2 SQL files for creating the database. license The directory contains the license for the Web application.
4.3.2 Web application installation details

The following steps discusses some important options that we selected when installing a secure Web application. Important: The security option is selected with the wizard. It is not a trivial task to change the security model once the Web application is installed. It is easier to uninstall and reinstall the Web application to enable security. Uninstallation needs to be performed from the /opt/IBM/ITMNP21/_uninst. The dialog options in the wizard are listed below: 1. Installation and configuration type dialog specifies the action and security authentication that is used. The installation action can be the Web application, language pack, or database; we choose to install the Web application and database. Remember that installing the database means any existing database will be destroyed. The security setting options are LDAP, Local OS, or None. We use LDAP security so that the user IDs using the WebSphere applications are authenticated to the LDAP registry. This allows us to use authentication from our z/OS RACF user registry. This option does not modify the WebSphere security option.
68
2. The database user dialog contains the database user ID, password, and database name to use. We simply use the instance owner db2inst1 and its password and use the name ITMNPDB for the database name. 3. Database communication dialog defines the port and loopback node. The port defines the port the DB2 instance is listening to. This port is typically 50000 and defined in the /etc/services file as the cDB2 service. The loopback node creation is necessary for WebSphere to access DB2 through the network communication. The DB2 Control Center panel will show two nodes: one local and one through TCP. We use the default name for the loopback node of ITMNPND. 4. Path to the WebSphere Application Server, typically /usr/WebSphere/AppServer. 5. WebSphere administrator user ID; typically this is wsadmin. Unless security is already enabled in the WebSphere, you can put anything in this panel. 6. Port numbers: You specify the port number for the itmnpUI application and the itmnpItsc application. See also 2.4.2, Communication port usage on page 35. The default secure ports are 9453 and 9454. The port relates to the next panel on configuring a secure communication. These ports need to be consistent with the content of itmnp.properties for monitors and nvexportd.properties for the ITSC interface. 7. WebSphere type: We install this in a base configuration, not at a Node Deployment version of WebSphere. 8. LDAP setting page: Specify the options for the LDAP server host name, user ID, and password. These are the parameters that the WebSphere will initially authenticate to the LDAP server. We set these to the SC61 z/OS system and use the TIVO01 user and password. 9. The installation path is where the distribution files will be installed; typically, it is /opt/IBM/ITMNP21. Then the installation will proceed.
4.3.3 Setting up LDAP in z/OS

In this section, we show how to configure the Lightweight Directory Access Protocol (LDAP) for z/OS LDAP server to run with the Security Database manager (SDBM) database. In order to use LDAP as the user registry, you need to have a valid administration user name, its password, the server host and port, the base DN, and, if necessary, the bind DN and the bind password. The user chosen can be any valid user in the registry and should be searchable. When security is enabled in the product, this server ID and password will be authenticated with the registry during the product startup. If authentication fails, the server will not come up.
69
If the WebSphere Application Server is configured to do LDAP authentication, then it can take advantage of native authentication. When the challenge to do LDAP authentication is presented, the user can enter the Security Server user ID and password (on the system where the LDAP server is running). The Web server will search the LDAP directory for an entry where the uid attribute equals the input user ID. The Web server will use the returned distinguished name (DN) and the password the user entered to do an LDAP bind operation. When the z/OS LDAP server determines that this entry is subject to native authentication, it will verify the password with the Security Server. The z/OS LDAP server accesses the LDAP Directory data in one of the two places: Normal LDAP directory data is stored in DB2 tables managed by the LDAP server. Initially, the server used an internal protocol called Relational Database Manager (RDBM) to access the directory data. With OS/390 V2R10, a new protocol called Traditional Database Manager (TDBM) has been introduced. It provides an optional alternative and gives better performance and flexibility. To talk to DB2, both TDBM and RDBM exploit the DB2 Call Level Interface (CLI). LDAP can also access data from selected RACF profiles kept in the RACF database, and therefore enable LDAP clients to indirectly exploit RACF. In this case, the RACF database is part of the LDAP directory, and an LDAP protocol called Security Database Manager (SDBM) handles the mapping of LDAP requests between LDAP and RACF. The structure of the LDAP directory entries must be defined in the schema files. IBM supplies a number of schema files (stored in UNIX HFS files) with LDAP, which support a general directory structure as well as a structure for entries required by IBM products exploiting LDAP. In this section, we configure a SDBM only LDAP directory server back end. LDAP on z/OS offers a configuration utility called ldapcnf to assist in the installation and customization of an z/OS LDAP server. To complete the installation process, follow the following steps: 1. Create an LDAP working directory for your new LDAP server. In our case, we use /etc/ldap. 2. Copy the sample profile from /usr/lpp/ldap/etc/ldap.profile to this new directory. In our case, this is /etc/ldap/ldap.profile. 3. Customize the ldap.profile file to reflect your system and the configuration variables by following the detailed descriptions of each attribute in the profile. Some attributes in the ldap.profile file are required, but not given a default value. Make sure you read through the entire file, completing all required variables. Although in our environment we only configure SDBM, you must
70
input dummy values into TDBM in order to successfully run the ldapcnf command. Our sample ldap.profile customization file is available in z/OS LDAP setup configuration file: ldap.profile on page 224. 4. Run ldapcnf from the UNIX System Services. This utility will generate a set of jobs in the MVS data set that were specified in the OUTPUT_DATASET definition in the ldap.profile. In our example, this is TIVO01.LDAP.OUTPUT. (See Example 4-3.)
Example 4-3 Output of ldapcnf utility TIVO03 @ SC62:/u/tivo03>/usr/lpp/ldap/sbin/ldapcnf -i /etc/ldap/ldap.profile The utility is finished checking for errors. Generating dbCli .... Finished generating dbCli. Generating dbSpufi .... Finished generating dbSpufi. Generating dsnaoini .... Finished generating dsnaoini. Generating ldapSrvProc .... Finished generating ldapSrvProc. Generating slapdcnf .... Finished generating slapdcnf. Generating irr .... Finished generating irr. Generating kerb .... Finished generating kerb. Generating slapdenv .... Finished generating slapdenv. Generating racf .... Finished generating racf. Generating prgmCtrl .... Finished generating prgmCtrl. Generating ocsfApf .... Finished generating ocsfApf. Generating ocsf .... Finished generating ocsf. Generating gldOcsfApf .... Finished generating gldOcsfApf. Generating PROGxx .... Finished generating PROGxx. Generating apf .... Finished generating apf. Exiting with return code 0.
5. Copy the LDAP server started task procedure from the output data set member to the system PROCLIB. We renamed the started task to LDAPSRV.
71
6. The following jobs are created in the output data set. Run each job in sequence. Check the output for successful return codes. a. RACF: This job defines and sets authorization for RACF profiles that are needed to run the LDAP server using the user ID called STC. You need to review and modify the appropriate definition that conforms to your security standard. b. APF: This job defines the APF authorization by activating the PROGxx definition from SDSF. You can also issue the SET PROG=xx command from the system console manually. c. PGRMCTRL: This is an optional job for setting program controlled profiles in RACF. When this facility is enabled, access to the LDAP libraries and other supporting libraries becomes restricted, and a user must be authorized to access these modules. 7. Comment off the definitions for TDBM-specific configuration settings in the SLDAPCNF member. Check to see if the suffix setting for SDBM_specific configuration settings is correct. Our example, we used Suffix o=itso and use the default for other settings. Our SLAPDCNF configuration file is available in z/OS LDAP configuration file: SLAPDCNF on page 225. 8. Start the LDAP server using the LDAPSRV started task. From SDSF, you can start the server by entering /S LDAPSRV. The LDAP server is ready when the following message appears in JOBLOG for LDAPSRV:
GLD0122I Slapd is ready for requests
9. To verify LDAP, the native LDAP commands are quite cumbersome to run from UNIX System Services. You may want to use a simpler way to access LDAP. We use an independently developed LDAP browser client, which can be downloaded from this URL:
http://www.iit.edu/~gawojar/ldap/
Follow the installation instructions for the prerequisites from the URL. Figure 4-8 on page 73 shows our connection setup for the LDAP browser to connect to our z/OS LDAP SDBM backend.
72
Figure 4-8 LDAP browser connection setup for SDBM back end
Figure 4-9 shows the LDAP browser connected to our z/OS LDAP SDBM back end, which is our RACF database. The profile type of the user, group, and connect are displayed.
Figure 4-9 LDAP browser to SDBM back end
10.Set up the user ID for the default WebSphere administrator to be authenticated through the z/OS LDAP. The default user WSADMIN needs to be defined in RACF. Note the password of the user ID so you can still access the Administrative Console after you enable WebSphere Security.
73
4.3.4 Configuring WebSphere Application Server for LDAP on z/OS

There are five steps to configuring the WebSphere Application Server to use the z/OS LDAP server. To perform these steps, log on to the WebSphere Administrator console as administrator. 1. Set up the authentication mechanism. The Light-weight Third Party Authentication (LTPA) protocol enables the WebSphere Application Server to provide security in a distributed environment using cryptography. This protocol uses cryptographic keys (LTPA keys) to encrypt and decrypt user data that passes between the servers. a. From the WebSphere Administration console, in the left-hand navigation tree, select Security Authentication Mechanisms LTPA. b. Complete the dialog; the password is a newly defined password that is used for LTPA encryption. You can use the default timeout of 120. c. Click the Generate Keys button to generate the keys. d. Click the OK button and the Save button on the message window followed by clicking on another Save button to save the master configuration. 2. Set up the LDAP connection. a. From the WebSphere Administration console, in the left-hand navigation tree, select Security User Registries LDAP. b. Complete the dialog, as shown in Figure 4-10 on page 75.
74
Figure 4-10 Values for the LDAP connection
c. We have to modify the Advanced LDAP setting on the LDAP User Registry configuration Additional Property. This ensures that the filtering settings for the Group Filter, User ID Map, and Group Number ID Map meet the authentication requirements of the users and groups of existing WebSphere Application Server applications and the z/OS LDAP SDBM back-end search filter. Figure 4-11 on page 76 shows the LDAP filters we used for the z/OS LDAP server SDBM backend. Refer to Chapter 18, Accessing RACF information, in z/OS V1R4.0 Security Server: LDAP Server Administration and Use, SC24-5923 for more details.
75
Figure 4-11 LDAP filter settings
d. Click OK, Save, and Save again. 3. Enabling global security with LDAP. a. From the WebSphere Administration console, in the left-hand navigation tree, select Security Global Security. b. Complete the dialog, as shown in Figure 4-12 on page 77.
76
Figure 4-12 Values for the LDAP global security settings
c. Click OK, Save, and Save again to save the master configuration. 4. Mapping users to roles. There are three security roles defined by the IBM Tivoli Monitoring for Network Performance application. The ITMOperator and ITMAdmin roles are used to define the operator and administrator roles respectively. Users must be mapped to one of these roles in order to access the Tivoli Monitoring for Network Performance user interface. The third role, ItscAllAuthority, is used by the monitor component when it connects to the Web application. It is also used to access configuration data and user preferences. a. From the WebSphere Administrative Console left-hand navigation, select Applications Enterprise Applications. b. Select the itmnp21 application and choose the property Map security roles to users/groups. c. Check the ITMAdmin option to add new users to the roles of the ITM administrator. d. Click Lookup users button to search for users to be added, as shown in Figure 4-13 on page 78. Select the users to be added and click OK.
77
Figure 4-13 Mapping users to roles for LDAP
e. Click OK, Save, and Save again to save the master configuration. 5. Recycling WebSphere Application Server. For WebSphere Application Server to recognize the changes we made to security in this section, we have to recycle WebSphere Application Server.
4.3.5 Verification of the distributed implementation

We verified our IBM Tivoli Monitoring for Network Performance Web application in a distributed environment in the following sections: Verification of Web application on page 79 Verification of IBM Tivoli Monitoring for Network Performance on page 80
78
Verification of Web application

We used a browser to access the WebSphere Application Server with HTTP protocol. We logged into the URL:
http://jakarta.itsc.austin.ibm.com:9091/admin
The WebSphere Application Server redirected the browser to a HTTPS URL as follows:
https://jakarta.itsc.austin.ibm.com:9043/admin/logon.jsp
We logged into the WebSphere Application Server admin console, as shown in Figure 4-14.
Figure 4-14 WebSphere Application Server secure logon console
We selected Applications Enterprise Applications to verify that itmnp21 is active (it should have a green arrow icon under the Status column), as shown in Figure 4-15 on page 80.
79
Figure 4-15 Verification of Web Application: itmnp21 is active
Verification of IBM Tivoli Monitoring for Network Performance

We used a browser to access the IBM Tivoli Monitoring for Network Performance with HTTP protocol. We logged into the URL:
http://jakarta.itsc.austin.ibm.com:9080/itmnp
The IBM Tivoli Monitoring for Network Performance redirected the browser to a HTTPS URL as follows:
https://jakarta.itsc.austin.ibm.com:9443/itmnp/login.jsp
However, the secure port is setup as 9453. We changed the login to 9453; otherwise, some functions may not work without accessing the correct port:
https://jakarta.itsc.austin.ibm.com:9453/itmnp/login.jsp
Figure 4-16 on page 81 shows the secure login panel of IBM Tivoli Monitoring for Network Performance.
80
Figure 4-16 IBM Tivoli Monitoring for Network Performance secure logon panel
Note: The following should be done after you install and start the monitors. After login, we verified that the Web application was connecting the monitors by selecting Manage Monitors Manage Monitor Configurations, as shown in Figure 4-17 on page 82.
81
Figure 4-17 Verification that the Web application and monitor are connected
4.4 Some problems and their solutions

In this section, we discuss the problems that we found during our Web application installation and solutions to those problems. The topics we discuss include: 4.4.1, Problem with console users on page 83 4.4.2, LDAP user ID character constraint on page 84 4.4.3, Uninstallation from admin console on page 84 4.4.4, Set up the X-windows DISPLAY properties to enable graphics on page 86 4.4.5, LTPA key generation problem on page 87
82
4.4.1 Problem with console users

We have selected LDAP for active user registry settings, and selected configure security transport to enable encryption and authentication between the monitors and the WebSphere Application Server. The installation works well when creating the database, but fails at the end with the message shown in Figure 4-18.
Figure 4-18 FNPI0352E error
We viewed our itmnp_install.log in /opt/IBM/ITMNP21/bin, which shows the explanation for the error shown in Example 4-4.
Example 4-4 Error showing AdminControl service not available 22: WASX7017E: Exception received while running file "/opt/IBM/ITMNP21/bin/itmnp.jacl"; exception information: com.ibm.ws.scripting.ScriptingException: AdminControl service not available
Do the following to make AdminControl service available: Create a wsadmin user ID on AIX, if it does not have one already, and set the password to the wsadmin password. This password was set during the Web application installation. Log on to the WebSphere Application Server admin console and select System Administration Console Users. If it does not have wsadmin and the LDAP Server User ID (in our case it was tivo01), add them with the proper roles. In our case, Figure 4-19 on page 84 shows our console users.
83
Figure 4-19 WebSphere Application Server console users
4.4.2 LDAP user ID character constraint

In the installation wizard, we specified the LDAP user ID and password and the LDAP server host name. User IDs with more than 12 characters will cause the following error message:
javax.jms.JMSException: MQJMS1092: userid must be less than 12 characters
In our case, the user ID without a good filter is:

racfid=tivo01,profiletype=user,o=itso
which exceeded 12 characters, hence the error. The LDAP character constraint was resolved when we used proper filtering for z/OS LDAP in WebSphere Application Server. See 4.3.4, Configuring WebSphere Application Server for LDAP on z/OS on page 74 for LDAP filter details. After changing filters, we were able to log in to the LDAP server using user ID tivo01.
4.4.3 Uninstallation from admin console

When you want to reinstall IBM Tivoli Monitoring for Network Performance, follow the steps in IBM Tivoli Monitor for Network Performance: Planning, Installing,
84
and Configuring, SC31-6363. As mentioned in the uninstallation steps, you have to run the ./uninstall_itmnp21aix.bin command from /opt/IBM/ITMNP21/_uninst. Do not uninstall the Web application from the WebSphere Application Server admin console. Uninstalling the Web application from the admin console will not run the uninstall jacl file in which the configuration changes are saved and you will get problems while reinstalling. When you reinstall the Web application, and if you have not properly uninstalled the Web application before, then you will see various errors. The errors are in message FNPI0002E; the following lines are the probable errors in itmnp_install.log:
FNPI0002E: FNPI0002E: FNPI0002E: FNPI0002E: WASQueue com.tivoli.ItscNotify is already defined ListenerPort ItscNotify is already defined JDBCProvider DB2 JDBC PROVIDER is already defined DataSource itmnpds is already defined
The above errors can be corrected by going to the WebSphere admin console and selecting the properties for those errors that already exist in the FNPI0002E message and deleting them manually. For example, we received the error shown in Example 4-5 about the existence of com.tivoli.ItscNotifyCF.
Example 4-5 error shown in itmnp_install.log while reinstalling the Web application 31: FNPI0002E: WASQueueConnectionFactory com.tivoli.ItscNotifyCF is already defined. 31: <<Exit configureWASQueueConnectionFactory>> 37: Exiting with RC=2 37: FNPI0109E: WebSphere Configuration Error
We corrected the error by opening the WebSphere Application Server admin console and selecting Resources WebSphere JMS Provider WebSphere Queue Connection Factories and deleting the Queue Connection Factory com.tivoli.itscNotifyCF from the screen in Figure 4-20 on page 86. Actually, you also need to delete com.tivoli.itmnp.RealtimeResponseCF.
85
Figure 4-20 WebSphere admin console - Queue Connection Factories properties
4.4.4 Set up the X-windows DISPLAY properties to enable graphics

On the AIX server, run export DISPLAY=AIXhost:0.0 and xhost + to make the AIX server available for graphics for all the clients that connect to this machine. If the x-server properties are not set properly, you could end up with the following message:
Error 500: Can't connect to X11 window server using ':0.0' as the value of the DISPLAY variable
as shown in Figure 4-21 on page 87.
86
Figure 4-21 Error 500 message
4.4.5 LTPA key generation problem

After installing the IBM Tivoli Monitoring for Network Performance Web application, we stopped the WebSphere Application Server and started it. We get an error in SystemOut.log, as shown in Example 4-5 on page 85.
Example 4-6 LTPA error [5/11/04 15:31:43:636 CDT] 35be67d5 LTPAServerObj E SECJ0238E: An unexpected exception occurred when trying to create the initial LTPAServerObject. The exception is java.lang.NullPointerException atcom.ibm.ws.security.ltpa.LTPAPrivateKey.decode(LTPAPrivateKey.java:57) at com.ibm.ws.security.ltpa.LTPAPrivateKey.<init>(LTPAPrivateKey.java:47) at com.ibm.ws.security.ltpa.LTPAServerObject.<init>(LTPAServerObject.java:252) at com.ibm.ws.security.ltpa.LTPAServerObject.initLTPAServer(LTPAServerObject.java: 183) at com.ibm.ws.security.ltpa.LTPAServerObject.getLTPAServer(LTPAServerObject.java:1 33) at com.ibm.ws.security.server.lm.ltpaLoginModule.initialize(ltpaLoginModule.java:1
87
29) at com.ibm.ws.security.common.auth.module.proxy.WSLoginModuleProxy.initialize(WSLo ginModuleProxy.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0...
To solve the error shown in Example 4-6 on page 87, we need to generate LTPA keys. To generate LTPA keys, we need the WebSphere admin console to be up. Since the WebSphere Application Server startup failed with an LTPA error, we need to disable the global security and then restart the server, which would now be in non-secure mode and generate the LTPA keys. To start WebSphere Application Server in non-secure mode, we just modified security.xml file under /usr/WebSphere/AppServer/config/cells/jakarta. We changed the value of the enabled variable to false. The contents of the original file are shown in Example 4-7, with the enabled variable highlighted.
Example 4-7 Modifying security.xml file /websphere/appserver/schemas/5.0/orb.securityprotocol.xmi" xmlns:security="http://www.ibm.com/websphere/appserver/schemas/5.0/security.xmi " xmi:id="Security_1" useLocalSecurityServer="true" useDomainQualifiedUserNames="false" enabled="true" cacheTimeout="1200" issuePermissionWarning="false" activeProtocol="BOTH" enforceJava2Security="false" activeAuthMechanism="LTPA_1" activeUserRegistry="LDAPUserRegistry_1" defaultSSLSettings="SSLConfig_1">
After the server is started in non-secure mode, use the WebSphere Administrative Console, in the left-hand navigation tree, to select Security Authentication Mechanisms LTPA and click on the Generate Keys button to generate the keys. See Figure 4-22 on page 89.
88
Figure 4-22 LTPA keys generation
4.4.6 Problem with SSL

We encountered the error SSLHandshakeException: unknown certificate in SystemOut.log, as shown in Example 4-8.
Example 4-8 SSL error [5/12/04 14:12:10:326 CDT] 5303a99 ItmnpJmxResou A ItmnpJmxResourceAdapterImpl. execute TRAS0014I: The following exception was logged javax.net.ssl.SSLHandshakeException: unknown certificate at com.ibm.jsse.bg.a(Unknown Source) at com.ibm.jsse.bg.startHandshake(Unknown Source) at com.ibm.net.ssl.www.protocol.https.b.n(Unknown Source) at com.ibm.net.ssl.www.protocol.https.p.connect(Unknown Source) at com.ibm.net.ssl.www.protocol.http.bw.getInputStream(Unknown Source) at com.ibm.net.ssl.www.protocol.http.bw.getHeaderField(Unknown Source) at com.ibm.net.ssl.www.protocol.http.bw.getResponseCode(Unknown Source) at com.ibm.net.ssl.internal.www.protocol.https.HttpsURLConnection.getResponseCode( Unknown Source) at
89
com.tivoli.itmnp.jmxconnector.ItmnpJmxResourceAdapterImpl.execute(ItmnpJmxResou rceAdapterImpl.java:180) at com.tivoli.itmnp.jmxconnector.ItmnpJmxInteraction.execute(ItmnpJmxInteraction.j ava:242) at com.tivoli.itmnp.connection.ejb.ItmnpMonitorCommandBean.executeCommand(ItmnpMon itorCommandBean.java:534)at com.tivoli.itmnp.connection.ejb.ItmnpMonitorCommandBean.issueCommand(ItmnpMonit orCommandBean.java:130)...
To solve the problem, we need to add the Java run-time properties shown in Example 4-9 to Application Servers server1 Process Definition Java Virtual Machine Generic JVM arguments from the WebSphere administrative console.
Example 4-9 Java SSL run-time properties -Djavax.net.ssl.keyStore= /usr/WebSphere/AppServer/etc/itmnpServerKeyStore.jks -Djavax.net.ssl.keyStorePassword=WebAS -Djavax.net.ssl.trustStore= /usr/WebSphere/AppServer/etc/itmnpServerTrustStore.jks -Djavax.net.ssl.trustStorePassword=WebAS
4.5 Start and stop procedures

This section discusses the start and stop procedures for the Web application. This process needs to be coordinated with various monitors running on z/OS systems. Since the monitors write data directly to DB2, it is better to stop the monitors before stopping the Web application server. The start and stop processes are performed as root. To start up all components, IBM Tivoli Monitoring for Network Performance provides a script called itmnp_was.sh (found under /opt/IBM/ITMNP21/bin). Manual startup can be performed by using the following procedure: 1. Source the environment for DB2 from /home/db2inst1/sqllib/db2profile; the command to do this is:
. /home/db2inst1/sqllib/db2profiles
2. Export the DISPLAY environment variable for the X-windows Java client to draw the charts under WebSphere:
export DISPLAY=jakarta.itsc.austin.ibm.com:0.0
3. Start the DB2 processes:

/home/db2inst1/sqllib/bin/db2start
90
4. Start the WebSphere Application Server:

/usr/WebSphere/AppServer/bin/startServer.sh
Stopping DB2 and WebSphere can be done by using the following commands: 1. To stop WebSphere Application Server, you need to specify the user and password when global security is on; otherwise, the stop command will be ignored:
/usr/WebSphere/AppServer/bin/stopServer.sh server1 -user TIVO01 -password xxxx
2. Source the DB2 environment:

. /home/db2inst1/sqllib/db2profiles
3. Check the application that accesses DB2; the application db2jcc is a monitor, db2bp is a DB2 command line processor, and Java is typically from WebSphere:
db2 list application
4. When no more applications are connected, you can stop db2 using the following command:
db2stop
or you can force stop DB2 and terminate all applications with the following command:
db2stop force
4.6 Backup and recovery

To avoid losing your data in case of a hardware or software failure or before some type of reinstallation, establish a routine for backing up your IBM Tivoli Monitoring for Network Performance installation. How frequently you back up your database depends on your organizations policy for backups. This section discusses how to back up the Web application in AIX environment. The following topics will be discussed: 4.6.1, File system backup on page 92 4.6.2, DB2 database backup on page 93 4.6.3, DB2 database restore on page 97 4.6.4, File system restore on page 98
91
4.6.1 File system backup

After you install the Web application, there are several file systems that are affected. You need to back up these file systems. We recommend you back up the following paths: /opt/IBM/ITMNP21: IBM Tivoli Monitoring for Network Performance files and installation information /usr/WebSphere/AppServer: WebSphere Application Server path that contains all the Web applications and its settings /home/db2inst1: DB2 instance files and database files There are several way to perform this backup on AIX systems, such as: The dumpvg command will dump the entire volume group. If all the file systems are residing in the same volume group, this is very effective. This typically dumps the volume group into a removable magnetic media, such as tape cartridges. The cpio command will copy files into tape device for storage and backup. The tar command will collect the files from a certain path and put them in a single file, which makes it easier to dump to tape or write to CD. You need to back up these files once after a successful installation of the IBM Tivoli Monitoring for Network Performance Web application. Unless there is a major upgrade or changes to the system, these backups will be sufficient. Note: Make sure that DB2 and WebSphere Application Server are stopped before trying to back up these directories. We created a Journal File System (JFS) using the smitty jfs command and mounted it to the directory /backup. We specified a size of 2 GB for the JFS. We used the tar command to collect the files from the paths in /backup:
tar -cvf /backup/itmnp21.tar /opt/IBM/ITMNP21 tar -cvf /backup/was.tar /usr/WebSphere/AppServer tar -cvf /backup/db2.tar /home/db2inst1
Then we can copy these files into a tape or CD.
92
4.6.2 DB2 database backup

There are two types of backup you can perform: Offline database backup. This type of backup requires an exclusive connection to the database, since all tablespaces in the database will be backed up. Online database backup. This type of backup is especially useful for users who run production databases and need to have one or more of the tablespaces in the database running continuously. When an online backup is used, only the tablespaces being backed up require an exclusive connection by the user. This permits other tablespaces within the database that are not being backed up to remain online for other applications to access. Note: If performing an online backup, make sure the roll forward recovery parameter logretain=ON or userexit is enabled in the database configuration file. An offline backup is required before any online backups can be performed. We intended to use the online backup. We need to perform the following steps: 1. Setting database configuration on page 93 2. Performing an offline backup on page 94 3. Listing the tablespaces on page 95 4. Performing an online backup on page 96
Setting database configuration

Since we use the instance owner, it has the SYSADM authority already. We check that both logretain and userexit are not set by issuing the db2 get db cfg command to display the configuration of the itmnpdb database, as shown in Example 4-10.
Example 4-10 Display of ITMNPDB configuration $ db2 get db cfg for itmnpdb Database Configuration for Database itmnpdb Database configuration release level Database release level . . . Backup pending = NO = 0x0a00 = 0x0a00
93
Database is consistent Rollforward pending Restore pending Multi-page file allocation enabled Log retain for recovery status User exit for logging status . . .
= NO = NO = NO = NO = NO = NO
We have to set the log retain for recovery status to YES by issuing the db2 update db cfg command, as shown in Example 4-11.
Example 4-11 Enabling logretain $ db2 update db cfg for itmnpdb using logretain on DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.
We need to recycle the database manager to pick the new changes by using the command in Example 4-12.
Example 4-12 Command to recycle DB2 $ db2stop force 06-01-2004 16:06:15 0 0 SQL1064N DB2STOP processing was successful. SQL1064N DB2STOP processing was successful. $ db2start 06-01-2004 16:07:52 0 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful
We verify that the changes have been updated by using the db2 get db cfg command. This indicates that there is a backup pending and will require a full offline backup before an online backup can be performed.
Performing an offline backup

We stop the monitors that are connected to the database and stop all applications connected to the database by using the command in Example 4-13. We have to verify that all applications have stopped.
Example 4-13 Command to stop all applications in database $ db2 force application all DB20000I The FORCE APPLICATION command completed successfully. DB21024I This command is asynchronous and may not be effective immediately. $ db2 list application
94
SQL1611W No data was returned by Database System Monitor.
SQLSTATE=00000
We perform the backup offline by using the command shown in Example 4-14.
Example 4-14 Offline backup command for DB2 $ db2 backup db itmnpdb to '/backup' Backup successful. The timestamp for this backup image is : 20040601161754
Listing the tablespaces

Before we can perform the online backup, we need to check the tablespace names by connecting to the database and issuing the list tablespaces, as shown in Example 4-15.
Example 4-15 List tablespaces for itmnpdb $ db2 connect to itmnpdb Database Connection Information Database server = DB2/6000 8.1.0 SQL authorization ID = DB2INST1 Local database alias = ITMNPDB $ db2 list tablespaces Tablespaces for Current Database Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: = = = = = 0 SYSCATSPACE System managed space Any data 0x0000
= = = = =
1 TEMPSPACE1 System managed space System Temporary data 0x0000
= = = = =
2 USERSPACE1 System managed space Any data 0x0000
95
Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal = = = = = 3 ITMNPTS1 System managed space Any data 0x0000
= = = = =
4 SD_TEMP2 System managed space Any data 0x0000
In Example 4-15 on page 95, the only tablespace that we need to back up is ITMNPTS1, which is a system managed space. The other tablespaces are either temporary (TEMPSPACE1 and SD_TEMP2), not used (USERSPACE1), or not changed (SYSCATSPACE). Tip: Tablespace SYSCATSPACE does change in terms of database statistic information, which is required to optimize DB2 performance. However, all this information can be regenerated using the UPDATE STATISTIC command.
Performing an online backup

We perform the online backup shown in Example 4-16.
Example 4-16 Online backup command for DB2 $ db2 backup db itmnpdb tablespace(itmnpts1) online to /backup Backup successful. The timestamp for this backup image is : 20040601163344
This online backup may need to be performed regularly based on a schedule. Another backup option is to use Tivoli Storage Manager (TSM), but we will not discuss that option here. This backup may also need to be moved to removable media (tape or CD) for recovery.
Creating a backup schedule

Online backup requires the restore process to have access to the database log. Offline backup is preferred in order to preserve database integrity. Typically, an
96
installation will perform a scheduled maintenance window and perform an offline backup. These offline backup files are offloaded and kept for full recovery purposes.
4.6.3 DB2 database restore

Before restoring a database, we need to have one of the following authorities on DB2: SYSADM SYSCTRL SYSMAINT If you are restoring to a new database (not an existing one), SYSADM or SYSCTRL is needed. An offline database restore will acquire an exclusive connection, so no application should be connected during this task. This implies that all monitors connected to this database must be shutdown. Also, make sure no other applications are connected to the database when performing an offline restore. We shut down the monitors in WTSC62 and WTSC66. We verify that there is no connection to the database and do the offline restore of the DB2 database, as shown in Example 4-17. Note that the backup timestamp is the timestamp from Example 4-14 on page 95.
Example 4-17 Offline restore command $ db2 restore db itmnpdb from /backup taken at 20040601161754 without rolling forward without prompting DB20000I The RESTORE DATABASE command completed successfully.
This command will restore the database image that is saved in /backup with a timestamp of 20040601161754 to the existing itmnpdb database. This command will effectively overwrite the old itmnpdb database files with the backed up database image files. The key phrase "without rolling forward" will keep the database manager from putting the restored database in the rollforward pending state (this phrase is not needed if the database was not enabled for rollforward recovery at the time it was backed up). If you are interested in recovering all database files to the point of the last successful transaction and the database was enabled for rollforward recovery at the time it was backed up, leave out the "without rolling forward" key phrase. We use the online restore command to restore the database, as shown in Example 4-18. Note also that the timestamp that we use is the timestamp from Example 4-16 on page 96.
97
Example 4-18 Online restore command $ db2 "restore db imtnpdb tablespace(itmnpts1) online from /backup taken at 20040601163344 without prompting" DB20000I The RESTORE DATABASE command completed successfully.
This command will restore the syscatspace and itmnpts1 tablespaces to the existing itmnpdb database while allowing the uninhibited processing of other tablespaces not involved in the online restore command. Again, the "taken at timestamp" key phrase must be declared if there is more than one backup image in the same folder. You can get the timestamps of the backup file by listing the file in the backup directory. As an example, the backup file called ITMNPDB.0.db2inst1.NODE0000.CATN0000.20040601161754.001 is a backup with a timestamp of 20040601161754.
4.6.4 File system restore

When the disk fails while moving to a new system, we need to restore the images that we have from the file systems images and database images. The following procedure can restore your AIX based IBM Tivoli Monitoring for Network Performance Web application: 1. Prepare the new machine, install AIX, and input all the required PTFs that are similar to the ones in the original server. 2. Prepare the system in a similar fashion to the installation preparation: a. Create the new file systems for /opt/IBM/ITMNP21, /usr/WebSphere/AppServer, and /home/db2inst1. b. Create groups db2grp1, mqm, and mqbrkrs. c. Connect root to mqm, mqbrkrs, and db2grp1. 3. Install WebSphere Application Server with JMS and DB2 with the required level similar to the original server. 4. Restore the backup from 4.6.1, File system backup on page 92 for /opt/IBM/ITMNP21, /usr/WebSphere/AppServer, and /home/db2inst1. 5. Restore the archived offline and online backup for DB2 and its log files:
tar -xvof /backup/was.tar tar -xvof /backup/db2.tar tar -xvof /backup/itmnp.tar
6. Perform an offline restore or online restore:

. /home/db2inst1/sqllib/db2profile
98
db2 restore db itmnpdb from /backup taken at 20040601161754 without rolling forward without prompting
99
100
Chapter 5.
Mainframe Web application implementation

This chapter documents our installation experiences for implementing IBM Tivoli Monitoring for Network Performance. Here we created a scenario where we set up a z/OS only environment with all the relevant components, using the IBM Tivoli Monitoring for Network Performance Monitor to monitor two z/OS systems. The following topics are covered in this chapter: 5.1, Scenario overview on page 102 5.2, Preparing for the implementation on page 103 5.3, Web application implementation on page 114 5.4, Start and stop procedures on page 117 5.5, Backup and recovery on page 119
101
5.1 Scenario overview

This chapter discusses the implementation of the IBM Tivoli Monitoring for Network Performance Web application on the z/OS system. The implemented components of our z/OS scenario are shown in Figure 5-1. We configured DB2, WebSphere Application Server, and the Web application to run on our SC61 system. We then configured the IBM Tivoli Monitoring for Network Performance monitors to run on SC61 and SC52 systems. We installed Tivoli Data Warehouse enablement pack so that our performance data, captured by IBM Tivoli Monitoring for Network Performance, could be summarized and migrated into the Tivoli Data Warehouse for archiving. We then used the Crystal Enterprise product, which was provided as part of Tivoli Data Warehouse, to generate historical reports.
z/OS WTSC61 Tivoli Data Ware house DB2 ITMNPDB DB2 TWH_CDW Netview Integrated TCP/IP Services Component z/OS WTSC52
Crystal Enterprise product
MQ (JMS) WMQX
Itmnp21 Web Appication
itmnpMonitor
WebSphere
EAS
Netview
itmnpMonitor
Figure 5-1 z/OS configuration
102
5.2 Preparing for the implementation

This section describes the tasks that need to be performed before you actually install IBM Tivoli Monitoring for Network Performance Web application. The tasks are: 5.2.1, Preparing HFS files on page 103 5.2.2, Preparing DB2 on page 104 5.2.3, Graphic package on page 108 5.2.4, Preparing WebSphere Application Server on page 110
5.2.1 Preparing HFS files

The Web application installation requires some files to be created in the systems. These file systems are implemented as Hierarchical File Systems (HFS) files. The required space for Web application implementation on UNIX System Services is as follows: /u/itmnp /tmp $WAS_HOME This is where the installation files will be loaded. We need 20 MB. The temporary file system needs to have at least 67 MB of free space. The WebSphere Application Server home directory that the Web application will be installed on needs to have at least 117 MB of free space.
Use the following commands to check the HFS requirements: Issue the UNIX System Services command df -k /u to show the amount of free space on the /u file system. Use the console command D OMVS,F to display the mounted file systems and their associated mount points. To allocate a new HFS dataset, use the TSO dataset allocate utility or the batch utility IEFBR14, as shown in Figure 5-2. HFS datasets are defined as TYPE=HFS and DSORG=PO.
//ALLOCHFS //* //IEFBR14 //DD1 //
JOB ACCT,USER,CLASS=A,NOTIFY=&SYSUID EXEC PGM=IEFBR14 DD DSN=OMVS.U.ITMNP.HFS,SPACE=(MB,(25,0)),DCB=(DSORG=PO), DSNTYPE=HFS,DISP=(,CATLG)
Figure 5-2 Defining an HFS dataset
Chapter 5. Mainframe Web application implementation
103
In order for a file system to be used in USS, it has to be mounted at a mount point as follows;
mount filesystem(OMVS.U.ITMNP.HFS) mountpoint(/u/itmnp) type(HFS) mode(RDWR)
We need to populate the /u/itmnp with the supplied tar files from /usr/lpp/itmnp/V2R1M0/web_app_inst/itmnp21zos.tar. The files in this path must be owned by the WebSphere administrator. The processing is shown in Figure 5-3. The default WebSphere administrator is WSADMIN with group WSCFG1.
TIVO01:/ >chown -R wsadmin:wscfg1 /u/itmnp TIVO01:/ >su - wsadmin WSADMIN:/u/wsadmin >cd /u/itmnp WSADMIN:/u/itmnp >tar -xvof /usr/lpp/itmnp/V2R1M0/web_app_inst/itmnp21zos.tar WSADMIN:/u/itmnp >chmod 755 *.sh Figure 5-3 Loading the z/OS tar file
5.2.2 Preparing DB2

We performed the steps needed to prepare for installing IBM Tivoli Monitoring for Network Performance Web application on z/OS, as described in IBM Tivoli Monitoring for Network Performance: Planning, Installation, and Configuration, SC31-6363. You need to find the correct DB2 HFS directory to use with IBM Tivoli Monitoring for Network Performance. If your installation has multiple versions of the DB2 HFS files, it is important to ensure that the code level of the DLLs in the DB2 HFS files matches the code level in z/OS dataset SDSNLOD2. The SDSNLOD2 library is included in the STEPLIB of the WebSphere Application Server controller and servant startup procedure. We encountered a problem where our HFS and SDSNLOD2 were mismatched. The error message was displayed in the WebSphere Application Servers servant SYSOUT, as shown in Figure 5-4.
./bborjtr.cpp+820 ... BBOO0220E CNTR0019E: Non-application exception occurred w "findByWildcardName". Exception data: java.lang.NoClassDefFoundError: COM/ibm/db2os390/sqlj/jdbc/DB2SQLJDriver Figure 5-4 JDBC error in WS5611S
To find the current level of DB2 code, we searched for the string DB2DriverInfo_native.C in member DSNAQLDA in our SDSNLOD2 dataset.
104
The maintenance level and compilation date is shown after this string. Figure 5-5 shows the result of our search. In our case, the level is pq84783.
/DB2DriverInfo_native.C level:pq84783 compiled:Feb 20 2004 20040220182156020A001 1 0 Figure 5-5 DB2DriverInfo in member DSNAQLDA
To find the DB2 HFS code level, execute the COM.ibm.db2os390.sqlj.util.DB2DriverInfo.class file in the db2j2classes.zip, as shown in Figure 5-6.
TIVO02:/u/tivo02: >cd /db2/db2v8/UQ86912/classes TIVO02:/db2/db2v8/UQ86912/classes: >ls DSNJDBC_JDBCProfile.ser db2jdbc.cursors IBM db2sqljjdbc.properties db2j2classes.zip db2sqljjdbc.properties.orig TIVO02:/db2/db2v8/UQ86912/classes: >export LIBPATH=../lib:$LIBPATH TIVO02:/db2/db2v8/UQ86912/classes: >java -cp ./db2j2classes.zip COM.ibm.db2os390.sqlj.util.DB2DriverInfo DB2DriverInfo main received an SQLException trying to retrieve the driver build version!! --> ** DB2 ERROR: DB2 for z/OS SQLJ/JDBC Driver level mismatch detected ** --> ** DB2 ERROR: Driver Classes build=DB2 8.1 UQ85385 JDBC 2.0 --> ** DB2 ERROR: Native DLL build=DB2 7.1 PQ69861 JDBC 2.0 Figure 5-6 Displaying the DB2 HFS level
The DB2 configuration component required the following steps: 1. Ensure that the Resource Recovery Service (RRS) is started and active in the z/OS system. This Started Task must be active before DB2 that DB2 uses the RRS facility. 2. Grant BIND permissions for DB2 using SPUFI. We used a user ID that has SYSADM privileges, so this step is not necessary. 3. Get the DB2 communication parameters. We obtained our DB2 location name and port number from the DB2 master address space message log. The location name and port number is displayed in the DSNL004I message, as shown in Figure 5-7 on page 106.
105
11.03.28 STC13266 DSNL004I 166 166 166 166 166 166
-DB8D DDF START COMPLETE 166 LOCATION DB8D LU USIBMSC.SCPDB8D GENERICLU -NONE DOMAIN -NONE TCPPORT 38030 RESPORT 38031
Figure 5-7 DB2 startup jes2 message log
4. Define the specified number of threads and batch connections. This is performed by modifying the sample installation job DSNTIJUZ. You need to check with the DB2 system programmer to find the latest source of this member. The DSNTIJUZ job modifies the initialization data module; this module is typically called DSNZPARM. The modified parameter is shown highlighted in Example 5-1. You need to restart DB2 to activate the changes.
Example 5-1 Sample excerpt of DSNTIJUZ . . . DSN6SYSP ACCUMACC=10, ACCUMUID=0, AUDITST=NO, BACKODUR=5, CHKFREQ=500000, CONDBAT=10000, CTHREAD=200, DBPROTCL=DRDA, DLDFREQ=5, DSSTIME=5, DSVCI=YES, EXTRAREQ=100, EXTRASRV=100, EXTSEC=YES, IDBACK=50, IDFORE=50, IDXBPOOL=BP0, IXQTY=0, LBACKOUT=AUTO, LOBVALA=10240, LOBVALS=2048, LOGAPSTG=100, MAXDBAT=200, MGEXTSZ=NO, MON=NO, MONSIZE=262144, PCLOSEN=5,
106
. . .
5. Increase the BP16K1 bufferpool size to 1000 buffers. IBM Tivoli Monitoring for Network Performance uses tablespaces that have a 16 KB page size; therefore, we need a 16 KB bufferpool. By default, DB2 already predefined some 16 KB bufferpools but with a size of 0. You can check the bufferpools using the command -DIS BPOOL from the DB2 interactive ISPF interface. We altered the BP16K1 buffer pool using the -DB8D ALT BP16K1(VPSIZE=1000) command. The bufferpool display now shows the presence of the BP16K1 buffer, as shown in Example 5-2.
Example 5-2 BP16K1 bufferpool display -DB8D DIS BUFFERPOOL(BP16K1) DSNB401I -DB8D BUFFERPOOL NAME BP16K1, BUFFERPOOL ID 121, USE COUNT 1 DSNB402I -DB8D BUFFER POOL SIZE = 1000 BUFFERS ALLOCATED = 1000 TO BE DELETED = 0 IN-USE/UPDATED = 0 BUFFERS ACTIVE = 20 DSNB406I -DB8D PGFIX ATTRIBUTE CURRENT = NO PENDING = NO PAGE STEALING METHOD = LRU DSNB404I -DB8D THRESHOLDS VP SEQUENTIAL = 80 DEFERRED WRITE = 30 VERTICAL DEFERRED WRT = 5, 0 PARALLEL SEQUENTIAL =50 ASSISTING PARALLEL SEQT= 0
6. Bind the JDBC packages using the job FNPLCBND from the FNPSAMP library. This will bind the DSNJDBC plan. You need to grant the execution of this plan for the user IDs that will run WebSphere or grant it to PUBLIC. 7. Edit and run the bindJDBC.sh command from UNIX System Services. This will bind the DB2 JDBC remote access module for the IBM Tivoli Monitoring for Network Performance monitors. Alternatively, from the DB2 HFS file system, you can issue the command shown in Example 5-3 to bind the DB2 JDBC remote access.
Example 5-3 JDBC bind command cd /usr/lpp/db2/db8d/jcc/classes java -cp ./db2jcc.jar:./db2jcc_license_cisuz.jar com.ibm.db2.jcc.DB2Binder -url jdbc:db2://wtsc61.itso.ibm.com:38030 -user TIVO01 -password xxxxxx
107
8. Create a storage group to assign dataset allocation parameters. We need to have the tablespaces for IBM Tivoli Monitoring for Network Performance created under the high level qualifier of DB8DU on disk TIVO01 and TIVO02. From SPUFI, we run the following SQL statement:
CREATE STOGROUP ITMNPSTO VOLUMES(TIVO01, TIVO02) VCAT DB8DU
9. Create the database tables using the job FNPCRDB from the FNPSAMP library. 10.Modify the file db2sqljjdbc.properties to reflect the correct DB2 properties. The file is in the classes sub-directory of the DB2 HFS path. Our modification of the file is shown in Example 5-4.
Example 5-4 Content of db2sqljjdbc.properties # Any lines starting with the pound sign '#' # are comments. Please see the DB2 for z/OS # Application Programming Guide and Reference # for Java for the description of these settings. # DB2SQLJSSID=DB8D db2.jdbc.profile.pathname=/usr/lpp/db2/db8d/classes/DSNJDBC_JDBCProfile.ser DB2SQLJPLANNAME=DSNJDBC DB2SQLJJDBCPROGRAM=DSNJDBC #DB2SQLJ_TRACE_FILENAME=/tmp/mytrc #DB2SQLJ_TRACE_BUFFERSIZE=1024
5.2.3 Graphic package

The z/OS environment does not have native X-Window support similar to the AIX environment. To generate the charts, the IBM Tivoli Monitoring for Network Performance Web application requires a graphic driver from the PJA Toolkit. The PJA Toolkit graphic driver can be retrieved from the Web address http://www.eteks.com/pja/en/#Download. The resulting Web site is shown in Figure 5-8 on page 109.
108
Figure 5-8 PJA Toolkit eteks download site
We download the toolkit for other platforms as a ZIP file and extract the file pja.jar. From our SC61 UNIX System Services, we create our pja install directories and copy the pja.jar, all font files, and FnpThonburi.ttf file, as shown in Figure 5-9. We use the path /u/itmnp/pja.
TIVO02:/u/itmnp: TIVO02:/u/itmnp: TIVO02:/u/itmnp: TIVO02:/u/itmnp: TIVO02:/u/itmnp: TIVO02:/u/itmnp: TIVO02:/u/itmnp:
>mkdir pja >mkdir /u/itmnp/pja/lib >mkdir /u/itmnp/pja/lib/fonts >cp /u/itmnp/FnpThonburi.ttf /u/itmnp/pja/lib/fonts >cp -r /usr/lpp/java/IBM/J1.3/lib/font* /u/itmnp/pja/lib >cp /tmp/pja.jar /u/itmnp/pja/lib/pja.jar >chmod -R 755 /u/itmnp/pja
Figure 5-9 Creating the pja environment
109
5.2.4 Preparing WebSphere Application Server

We install our WebSphere Application Server on the WTSC61 system, which includes the Java Messaging Services (JMS) component. Java Messaging Services is an embedded messaging component that forms part of WebSphere Application Server. An alternative to using this component is to use the WebSphere MQ product. The tasks that need to be performed on WebSphere Application Server are: Allocating DB2 libraries to WebSphere on page 110 Modifying WebSphere settings on page 111 Installing a sample graphic test program on page 112
Our WebSphere environment

In our environment, once the WebSphere Application Server starts, the following started tasks are active: WS611 - WebSphere Application Server control region WS611S - WebSphere Application Server servant WS611D - WebSphere Application Server daemon WMQXMSTR - JMS queue manager WMQXCHIN - JMS channel initiator The WebSphere Application Server is active when the BBOO0222I message, as shown in Figure 5-10, is displayed in the servants JES2 message log.
BBOO0222I WSVR0001I: Server SERVANT PROCESS ws611sc61 open for e-business BBOO0020I INITIALIZATION COMPLETE FOR WEBSPHERE FOR Z/OS SERVANT PROCESS WS611. Figure 5-10 WebSphere Application Server active message
We use the WebSphere Administrative Console to configure the WebSphere variables and enable global security. The SYSPRINT for the WebSphere Application Server servant region address space contains the problem logs for WebSphere applications. It is important to view this log file when troubleshooting WebSphere Application Server or WebSphere application problems.
Allocating DB2 libraries to WebSphere

The DB2 libraries have to be accessible to the WebSphere Application Server control and servant processes. A STEPLIB (or LINKLIST entry) has to be added
110
to the WebSphere Application Server control STC and WebSphere Application Server servant STC for the following datasets: SDSNEXIT SDSNLOAD SDSNLOD2
Modifying WebSphere settings

These WebSphere settings are modified from the WebSphere Administrative Console. The variables that we modified are summarized in Table 5-1. A more detailed procedure is in IBM Tivoli Monitoring for Network Performance: Planning, Installation, and Configuration, SC31-6363.
Table 5-1 WebSphere setting modification Menu path Environment Manage WebSphere Variables Servers Application Servers ws611 Servers Application Servers ws611 Custom Properties Variable name DB2390_JDBC_DRIVER_PATH DB2SQLJPROPERTIES Total transaction lifetime timeout Maximum transaction timeout protocol_http_timeout_output_ recovery protocol_https_timeout_output_ recovery ConnectionIOTimeout ConnectionResponseTimeout Our value /usr/lpp/db2/db8d /usr/lpp/db2/db8d/classes/db2sqljjdbc. properties 3600 4200 SESSION SESSION 3600 2400
Servers Application Servers ws611 Web Container HTTP Transport 9080 Custom Properties Servers Application Servers ws611 Web Container HTTP Transport 9443 Custom Properties Servers Application Servers ws611 ORB Service Advanced Setting
ConnectionIOTimeout ConnectionResponseTimeout
3600 2400
WLM Timeout
2400
111
Menu path Servers Application Servers ws611 Process Definition Servant Java Virtual Machine Servers Application Servers ws611 Process Definition Servant Java Virtual Machine Custom Properties
Variable name GenericJVMArguments
Our value -Xbootclasspath/p:/u/itmnp/pja/lib/pja.jar
awt.toolkit java.awt.fonts java.awt.graphicsenv java2d.font.usePlatformFont user.home user.timezone
com.eteks.awt.PJAToolkit /u/itmnp/pja/lib/fonts com.eteks.java2d.PJAGraphics Environment true /u/itmnp/pja CST Checked Not checked 1200 SWAM LocalOK Not checked true
Security Global Security
Enabled Enforce Java 2 Security Cache timeout Active Authentication Active User Registry
Security Global Security Custom Properties
com.ibm.security.useFIPS EnableTrustedApplications
Security User com.ibm.security.SAF.authorization true Registries LocalOS false Additional Properties com.ibm.security.SAF.EJBROLE. Audit.Messages.Supress Custom Properties
After all the variables has been entered, you can click on Save and Save again to modify the Master Configuration.
Installing a sample graphic test program

The graphic test program is supplied in an ear (Enterprise Application Archive) file. 1. From the administrative console, select Application Install New Application. Install the application from the /u/itmnp/jctest.ear file, and accept all the default values.
112
2. Select Application Enterprise Application and click on jctest, and click on the Configuration tab. Set the following: Classloader Mode PARENT_LAST WAR Classloader Policy Application 3. Click OK and Save and Save again. 4. Restart the WebSphere Application Server. 5. Back on the Administrative Console, select Application Enterprise Application, check the box beside jctest, and click Start. 6. We ran the jctest application by opening a browser and entering the following URL:
http://wtsc61.itso.ibm.com:9080/jctest/
The result was the display of the Graphing Setup Test graph, as shown in Figure 5-11. At this stage, we are sure that our graphics driver is working.
Figure 5-11 Jctest graph
113
5.3 Web application implementation

The Web application implementation in this section discusses the problems and experiences we encountered while doing this installation.
5.3.1 Installation procedure overview

The implementation steps on z/OS can be summarized in the following steps: 1. Generate the itmnp.env file. When you run the command /u/itmnp/itmnp.sh install the first time, it will generate the itmnp.env file. This file contains the setting that will be used for installing the Web application. 2. Edit the itmnp.env file. The parameters in the itmnp.env file are explained in great detail in IBM Tivoli Monitoring for Network Performance: Planning, Installation, and Configuration, SC31-6363. A sample for our installation is given in Example 5-5.
Example 5-5 Out itmnp.env file # v2.1.0.0.57 # For information about setting the values below, refer to the IBM Tivoli # Monitoring for Network Performance: Planning, Installation, and Configuration # Guide chapter about installing the Web application on z/OS. # DB2USER=TIVO01 DB2PASSWD=TIVO01 UI_PORT=9080 MONITOR_PORT=9081 CONFIG_FOR_SECURE_MONITOR=N ACTIVE_USER_REGISTRY_CONFIGURATION=LOCALOS LDAP_SERVER_FQDN_HOST=ldap.myserver.net.com LDAP_SERVERID=cn=root,o=ibm,c=us LDAP_SERVERPW=mypassword WASADMIN_USERNAME=wsadmin WASADMIN_PASSWORD=wsadmin ITMNP_DB_LOCATION_NAME=DB8D ITMNP_DB_TYPE=DB2UDBOS390_V7 NODE= SERVER=ws611sc61 TRACE_INSTALL=OFF WAS_50_BIN=/SC61/WebSphereBD/V5R0M0/BS01/AppServer/bin
3. Test your configuration using the command /u/itmnp/itmnp.sh test.
114
4. Change to superuser using the su command and run the command /u/itmnp/itmnp.sh install again to actually configure WebSphere Application Server and install the itmnp21 application. 5. Restart WebSphere and log into the administrative console. Go to Application Enterprise Application and check the status of the itmnp21 application. The status should be green (started). 6. Create the necessary RACF profiles for the EJBROLE class. The commands to do this are shown in Example 5-6.
Example 5-6 Defining EJBROLE SETROPTS CLASSACT(EJBROLE) RDEFINE EJBROLE ItscAllAuthority UACC(READ) RDEFINE EJBROLE ITMOperator UACC(NONE) RDEFINE EJBROLE ITMAdmin UACC(NONE) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO01) ACC(READ) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO02) ACC(READ) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO03) ACC(READ) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO04) ACC(READ) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO05) ACC(READ) PERMIT ITMAdmin CLASS(EJBROLE) ID(TIVO06) ACC(READ) PERMIT ITMOperator CLASS(EJBROLE) ID(ITMNPOP) ACC(READ)
5.3.2 Verification
We verified that our IBM Tivoli Monitoring for Network Performance Web Application installed correctly by going to the WebSphere Administrative Console and selecting Applications Enterprise Applications. Here we saw that our itmnp21 application has a green arrow beside it, as shown in Figure 5-12 on page 116. This was an indication that our installation was successful.
115
Figure 5-12 Web Application installation result
To verify that the IBM Tivoli Monitoring for Network Performance Web application was working, we did the following: We started a browser session and entered the following URL:
http://wtsc61oe.itso.ibm.com:9080/itmnp
The welcome screen was then displayed, as shown in Figure 5-13 on page 117.
116
Figure 5-13 Web application welcome screen

This chapter discusses the start and stop procedure for IBM Tivoli Monitoring for Network Performance Web application environment. The monitors start and stop process should be coordinated with this process.
5.4.1 Start up the Web application

IBM Tivoli Monitoring for Network Performance should be started in the following sequence: 1. DB2 has to be started as follows:
-db2_system_name start db2
117
2. Start the WebSphere Application Server from the system console as follows:
S controlregionprocname, JOBNAME=server_shortname, ENV=cell_shortname.node_shortname.server_shortname
where: controlregionprocname The JCL procedure name in PROCLIB. server_shortname The stepname used to start the procedure. Cell_shortname.Node_shortname.Server_shortname The ENV variable is a concantenation of the cell shortname, the node shortname, and the server shortname. In our environment, we started WebSphere Application Server, as shown in Figure 5-14.
S WS5611C,JOBNAME=WS611,ENV=CL611.ND611.WS611 IRR812I PROFILE WS5611C.* (G) IN THE STARTED CLASS WAS USED TO START WS5611C WITH JOBNAME WS611. $HASP100 WS611 ON STCINRDR IEF695I START WS5611C WITH JOBNAME WS611 IS ASSIGNED TO USER ASCR1, GROUP WSCFG1 $HASP373 WS611 STARTED IEF403I WS611 - STARTED - TIME=14.26.59 - ASID=0081 - SC61 BBOO0001I WEBSPHERE FOR Z/OS CONTROL PROCESS CL611/ND611/CLU611/WS611 IS STARTING. Figure 5-14 SC61 WebSphere Application Server startup
The controller region automatically starts the daemon. This is the necessary command:
START WS5611D,JOBNAME=WS611D,ENV=CL611.CL611.WS611D
The JMS queue manager is then started automatically with the command:
S WMQXMSTR,SUFFIX=R
The JMS channel initiator also starts automatically with the command:
S WMQXCHIN
Lastly, the servant is started automatically by Workload Manager (WLM) with the command:
START WS5611S, JOBNAME=WS611S,ENV=CL611.ND611.WS611
118
5.4.2 Shut down the Web application

IBM Tivoli Monitoring for Network Performance should be stopped in the following sequence: 1. Stop the WebSphere Application Server from the system console as follows:
P WS611
where WS611 is the control region step name of the WebSphere Application Server. The JMS queue manager and JMS channel initiator will also be stopped as a result of this command. The JMS queue manager is stopped automatically by the control region, such as by using the command P WMQXMSTR. The JMS channel initiator is also stopped automatically using the command P WMQXCHIN. Lastly, the servant is stopped automatically by WLM with the command P WS611S. 2. Stop the WeSphere Application Server daemon as follows:
P WS611D
where WS611D is the control region daemon name of the WebSphere Application Server. 3. Stop DB2. You need to ensure all remote threads have disconnected from the output of the DIS THREAD(*) command. The command to stop DB2 is:
-DB8D STOP DB2 MODE(QUIESCE)
The command prefix in our environment is -DB8D.
5.5 Backup and recovery

The IBM Tivoli Monitoring for Network Performance database can become unusable because of hardware or software failure or both. A routine for backing up your Tivoli Monitoring for Network Performance database has to be established. How frequently you back up your database depends on how frequently you make changes to your Tivoli Monitoring for Network Performance installation, and on your organizations policies for backups. The WebSphere Application Server, IBM Tivoli Monitoring for Network Performance Web application, and IBM Tivoli Monitoring for Network Performance monitor contain HFS file systems that also require backing up. HFS file systems can be backed up using utilities, such as the ADRDSSU(DFDSS) dump facility, or UNIX shell commands, such as pax cpio and tar.
119
5.5.1 Backup and restore of file systems

There are several HFS files that you may want to back up that are related to the IBM Tivoli Monitoring for Network Performance Web application; they are: WebSphere Application Server configuration IBM Tivoli Monitoring for Network Performance install directory We back up our HFS using the ADRDSSU utility. Before ADRDSSU can successfully DUMP the HFS, the file systems must be unmounted using the UMOUNT command from TSO. The ADRDSSU job is shown in Figure 5-15.
//DUMPHFS JOB (ACCOUNT),'TIVO02',CLASS=A,NOTIFY=&SYSUID /*JOBPARM S=SC61 //STEP1 EXEC PGM=ADRDSSU,REGION=2048K //SYSPRINT DD SYSOUT=* //OUTDD1 DD DSN=VBUDI.WAS61.DUMP, // SPACE=(CYL,(40,20)), // DISP=(NEW,CATLG,DELETE), // UNIT=3390, // VOL=(,,,,SER=TIVO02) //SYSIN DD * DUMP DATASET( INCLUDE( OMVS.SC61.WAS5BD.BS01.CONFIG.HFS )) OUTDDNAME(OUTDD1) TOL(ENQF) /* Figure 5-15 DFDSS HFS dump
To restore the HFS dataset, we ran our restore job, which is shown in Figure 5-16 on page 121. This restore job replaced the existing WebSphere Application Server HFS dataset with the backed up dataset. Note that the target HFS file must not be mounted for the restore process to succeed.
120
//RESTHFS JOB (ACCOUNT),'TIVO02',CLASS=A,NOTIFY=&SYSUID /*JOBPARM S=SC61 //STEP1 EXEC PGM=ADRDSSU,REGION=2048K //SYSPRINT DD SYSOUT=* //INDD1 DD DSN=VBUDI.WAS61.DUMP, // LABEL=(1,SL), // DISP=(OLD,KEEP,KEEP) //SYSIN DD * RESTORE DATASET( INCLUDE( OMVS.SC61.WAS5BD.BS01.CONFIG.HFS )) INDDNAME(INDD1) CANCELERROR REPLACE CATALOG TGTALLOC(SOURCE) WAIT(2,2) /* Figure 5-16 DFDSS HFS restore
After the backup or restore process completes, we need to re-mount the HFS dataset. We issue the following mount command to mount the HFS.
mount filesystem(OMVS.SC61.WAS5BD.BS01.CONFIG.HFS) mountpoint(/SC61/WebSphereBD/V5R0M0/BS01) type(HFS) mount filesystem(OMVS.U.ITMNP.HFS) mountpoint(/u/itmnp) type(HFS)
5.5.2 Backup and restore of DB2 database

We used the copy utility to create a full image copy of our ITMNP DB2 tablespaces. To display the ITMNPDB tablespace names, we issued the DB2 summary display command for our ITMNP database, as shown in Figure 5-17 on page 122.
121
-DB8D DIS DB(ITMNPDB) DSNT360I -DB8D *********************************** DSNT361I -DB8D * DISPLAY DATABASE SUMMARY * GLOBAL DSNT360I -DB8D *********************************** DSNT362I -DB8D DATABASE = ITMNPDB STATUS = RW DBD LENGTH = 165548 DSNT397I -DB8D NAME TYPE PART STATUS PHYERRLO PHYERRHI CATALOG PIECE -------- ---- ----- ----------------- -------- -------- -------- ----WCPT2 TS RW WCPT3 TS RW WCPTS TS RW BP1 IX RW . . . Figure 5-17 Displaying ITMNP database
We then added these tablespace names to our backup job, as shown in Figure 5-18.
//DB2COPY JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID,CLASS=A,MSGCLASS=T, // MSGLEVEL=(1,1),REGION=4096K,TIME=1440 /*JOBPARM SYSAFF=SC61,L=9999 //UTIL EXEC DSNUPROC,SYSTEM=DB8D,UID='TEMP',UTPROC='' //DSNUPROC.WCPT2 DD DSN=TIVO01.ITMNP.WCPT2.BKP0106,DISP=(,CATLG), // SPACE=(CYL,(250,20),,,ROUND),UNIT=3390,VOL=SER=TIVO04 //DSNUPROC.WCPT3 DD DSN=TIVO01.ITMNP.WCPT3.BKP0106,DISP=(,CATLG), // SPACE=(CYL,(250,20),,,ROUND),UNIT=3390,VOL=SER=TIVO04 //DSNUPROC.WCPTS DD DSN=TIVO01.ITMNP.WCPTS.BKP0106,DISP=(,CATLG), // SPACE=(CYL,(250,20),,,ROUND),UNIT=3390,VOL=SER=TIVO04 //DSNUPROC.SYSIN DD * COPY TABLESPACE ITMNPDB.WCPT2 COPYDDN(WCPT2) TABLESPACE ITMNPDB.WCPT3 COPYDDN(WCPT3) TABLESPACE ITMNPDB.WCPTS COPYDDN(WCPTS) // Figure 5-18 DB2 ITMNP backup job
To recover these table spaces from the backup we used before, we used the job shown in Figure 5-19 on page 123.
122
//DB2REST JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID,CLASS=A,MSGCLASS=T, // MSGLEVEL=(1,1),REGION=4096K,TIME=1440 /*JOBPARM SYSAFF=SC61,L=9999 //UTIL EXEC DSNUPROC,SYSTEM=DB8D,UID='TEMP',UTPROC='' //DSNUPROC.WCPT2 DD DSN=TIVO01.ITMNP.WCPT2.BKP0106,DISP=SHR //DSNUPROC.WCPT3 DD DSN=TIVO01.ITMNP.WCPT3.BKP0106,DISP=SHR //DSNUPROC.WCPTS DD DSN=TIVO01.ITMNP.WCPTS.BKP0106,DISP=SHR //DSNUPROC.SYSIN DD * RECOVER TABLESPACE ITMNPDB.WCPT2 RECOVERYDDN(WCPT2) TABLESPACE ITMNPDB.WCPT3 RECOVERYDDN(WCPT3) TABLESPACE ITMNPDB.WCPTS RECOVERYDDN(WCPTS) // Figure 5-19 DB2 ITMNP recover job
Your DB2 database administrator needs to be consulted for a more comprehensive backup and restore procedure for the database.
123
124
Chapter 6.
Monitor implementation and operation

This chapter discusses the implementation of the primary function of IBM Tivoli Monitoring for Network Performance, the monitors. Monitors are agent programs that run on each z/OS systems and collect information on TCP/IP network performance. The discussion in this section is divided into: 6.1, Monitor installation process on page 126 6.2, Some problems and their solutions on page 130 6.3, Start and stop procedures on page 132 6.4, Sample monitor configuration on page 134 6.5, Monitoring best practices on page 137
125
6.1 Monitor installation process

This section discusses the installation of the IBM Tivoli Monitoring for Network Performance monitor. We configured the IBM Tivoli Monitoring for Network Performance monitors based on the instructions in IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363 to perform the monitor implementation. The monitor component must be installed on each z/OS system you want to monitor. The discussion is divided into the following topics: 6.1.1, Before the installation on page 126 6.1.2, Preparing the system on page 127 6.1.3, Parameters for itmnp.properties on page 129
6.1.1 Before the installation

The monitor code is installed, using the System Modification Program/Extended (SMP/E) process, into a UNIX System Services directory. The monitor uses several directory paths for its execution, as discussed in 2.3, Monitor functions on page 22. Those directories are listed in Table 6-1.
Table 6-1 Path used by the monitor Function SMP/E target libraries Common directory setting Tivoli common log directory Configuration files Database cache files Default value /usr/lpp/itmnp/V2R1M0 /etc/ibm/tivoli/common/cfg/log.prop erties /var/ibm/tivoli/common/FNP/logs /etc/itmnp /var/FNP/dbcache Defined from DDDEFs FNPlogPropertiesLocation in itmnpMonitor.sh log.properties CONFIG_DIR in itmnpMonitor.sh itmnp.properties
We felt that we need to coordinate these files into a discrete set of paths so that the maintenance of HFS datasets are easier. We decided to leave the SMP/E installation alone and merge the following: Put the database cache files into the common log directory and create an HFS file for /var/ibm/tivoli/common/FNP with two director logs and dbcache underneath it. We allocate 120 MB on this file system. Consolidate the configuration files under the common directory structure /etc/ibm/tivoli/common with cfg and itmnp underneath it. We allocate 10 MB on this file system.
126
The resulting HFS files that we use are shown in Table 6-2.
Table 6-2 HFS allocation HFS file OMVS.ITMNP21.HFS OMVS.FNP.LOGS.HFS OMVS.TIVOLI.CFG.HFS Size xx MB 120 MB 10 MB Mount point /usr/lpp/itmnp /var/ibm/tivoli/common/FNP /etc/ibm/tivoli/common Linked to /itmnp $SYSNAME/FNP $SYSNAME/tivoli
Note that the common directories are system specific in the SYSPLEX, while the SMP/E installation HFS are shared across SYSPLEX images.
6.1.2 Preparing the system

There are some setup actions that must be performed on the following items: TCP/IP address space on page 127 OSNMPD process on page 127 Figure on page 128 RACF security profiles on page 128 Post installation setup on page 128
TCP/IP address space

There are several tasks to perform for the general Communication Server processes that we need to modify. They are: Enabling the TCP/IP address space for monitoring by adding a NETMONITOR statement to our TCP/IP profile. This definition can also be activated using the V TCPIP,,OBEY,dataset command from the system console:
NETMON SMFS TCPCONNS MINLIFET 0
Autolog OSNMPD in the AUTOLOG section of TCP/IP profile. SNMP data collection is important for some performance information and z/OS identification for NetView ITSC. Enabled SNAMGMT in VTAM using the command F NET,VTAMOPTS,SNAMGMT=YES.
OSNMPD process
The OSNMPD daemon can be autostarted by creating an AUTOLOG entry in the TCP/IP profile definition. This requires the existence of the OSNMPD started task to be defined on the system. Some of the TCP/IP parameter metrics are retrieved
Chapter 6. Monitor implementation and operation
127
from SNMP. We found that in our environment, the SNMP port 161 is reserved for the SNMPD started task instead of OSNMPD. We also need to change this parameter in the PORT section. We use the obey file shown in Example 6-1.
Example 6-1 Deleting SNMP port DELETE PORT 0161 UDP SNMPD DELETE PORT 0162 UDP SNMPQE
Activate it using the command V TCPIP,,OBEY,dataset.
UNIX System Services parameters

To optimize the performance of IBM Tivoli Monitoring for Network Performance in the UNIX System Services, some parameters in the USS need to be changed. These parameters reside in the BPXPRMnn member of SYS1.PARMLIB and are loaded when the USS is started. These are the list of parameter changes: MAXASSIZE MAXFILEPROC MAXTHREADS MAXPROCSYS MAXPROCUSER 1073741824. The Java environment requires that this limit must be high enough. 1000. 10000. 200. 25.
MAXTHREADTASKS 5000.
RACF security profiles

There are some security profiles that RACF can use. Run job FNPDEFID to define users for the IBM Tivoli Monitoring for Network Performance monitor component. Run job FNPUAUTH to define access for a monitor user ID to the z/OS communications server network management application programming interfaces.
Post installation setup

Run the itmnpPostMon.sh or perform the following manually: Set owner, authorization bit, and setuid fields for the distribution files Copy itmnp.properties and jks files to /etc/itmnp Create the log.properties file that point to the tivoli common directory Check the INSTALL_DIR, CONFIG_DIR, and JAVA_HOME variables in itmnpMonitor.sh. They should point to the correct paths.
128
6.1.3 Parameters for itmnp.properties

The primary configuration for the monitor resides in the itmnp.properties files. We copied the properties file and digital certificate files into the /etc/itmnp directory from /opt/IBM/ITMNP21/config. Changes to the itmnp.properties file that we performed are shown in Table 6-3. The table merged the changes that we perform for z/OS and the AIX Web application. SC61 connects to z/OS Web application and SC62 connects to the AIX Web application. Added our monitor_hostname as the fully qualified host name wtsc62.itso.ibm.com. Defaulted local_httpport to 9455 for communication between WebSphere Application Server and the monitor. The monitor listens on port 9455. Added the fully qualified host name of the server running our WebSphere Application Server to WAS_hostname. In our case, it was wtsc62.itso.ibm.com. Defaulted WAS_httpport to 9454 for communication between the monitor and WebSphere Application Server. WebSphere Application Server listens on port 9454. Added DBName system name ITMNPDB, as defined in the Web application wizard. Added a valid DB2 user name, password, and host name values. Added DBPort value of 50000, as defined in the Web application wizard.
Table 6-3 The itmnp.properties changes Parameter name monitor_hostname bind_interface local_httpport WAS_hostname WAS_httpport socksServer socksPort DBName DBUserName AIX Web application wtsc62.itso.ibm.com 0.0.0.0 9455 jakarta.itsc.austin.ibm.com 9454 Not used Not used ITMNPDB db2inst1 z/OS Web application wtsc61.itso.ibm.com 0.0.0.0 9081 wtsc61.itso.ibm.com 9080 Not used Not used DB8D TIVO01
129
Parameter name DBPassword DBHostName DBPort DBDriverType DBCacheDirectory DBCacheRowLimit enableCloudscape db2Security kerberosServer WebSphereServletProtocol trustStoreName trustStorePassword keyStoreName keyStorePassword keyManagerPassword CSAPIport monitor.trace.level
AIX Web application ###### jakarta.itsc.austin.ibm.com 50000 UNIVERSAL /var/ibm/tivoli/common/FNP Used default CLEAR_TEXT_PASSWORD_ SECURITY HTTPS itmnpMonitorTrustStore.jks WebAS itmnpMonitorKeyStore.jks WebAS ###### 1670 DEBUG_MIN
z/OS Web application ###### wtsc61.itso.ibm.com 38030 (default) /var/ibm/tivoli/common/FNP Used default CLEAR_TEXT_PASSWORD_ SECURITY HTTP 1670 DEBUG_MIN
Unlike the AIX implementation, the z/OS implementation does not include security between the monitor and the WebSphere Application Server. The communication transport between these two components are not authenticated and not encrypted in z/OS. Authentication and encryption requires LDAP authentication through WebSphere Application Server, and is only available when running on an AIX system. The parameters regarding security are not applicable in the z/OS implementation.
6.2 Some problems and their solutions

This section collects some problems that we encounter during the monitor implementation. 6.2.1, Missing Tivoli common directory on page 131
130
6.2.2, Setting APF authorized attribute on page 131
6.2.1 Missing Tivoli common directory

When installing the monitor on WTSC62, we received the message FNPM012E, as shown in Figure 6-1. The monitor did not find the Tivoli common directory specified, which should be defined in the log.properties file. To rectify this problem, we added the tivoli_common_dir=/var/tivoli/common statement to the /etc/ibm/tivoli/common/cfg/log.properties file.
+FNPM012E THE TIVOLI COMMON DIRECTORY WAS NOT FOUND IN THE /etc/ibm/ti voli/common/cfg/log.properties FILE; THE DEFAULT VALUE, /var/ibm/tivol i/common, WILL BE USED. Figure 6-1 Monitor Tivoli common directory not found message
6.2.2 Setting APF authorized attribute

Some programs and files associated with IBM Tivoli Monitoring for Network Performance need to be authorized. In our environment, the monitor was supplied as a tar file. The APF extended attributes are lost during the tar or copy process and have to be set again. We changed the attributes for the files /usr/lpp/itmnp/V2R1M0/bin and /usr/lpp/itmnp/V2R1M0/lib directories using the command:
extattr +a bin/monitor.cs390 lib/lib*
Figure 6-2 on page 132 shows what the attribute displays should look like using the ls -lE command. The second column of the display should show a-s-.
131
TIVO02:/: >cd /usr/lpp/itmnp/V2R1M0/bin TIVO02:/itmnp/V2R1M0/bin: >ls -lE monitor_cs390 -rwxr-x--- a-s- 1 AAAAAAA SYS1 1839104 Apr TIVO02:/itmnp/V2R1M0/bin: > TIVO02:/itmnp/V2R1M0/config: >cd ../lib TIVO02:/itmnp/V2R1M0/lib: >ls -lE lib* -rwxr-x--- a-s- 1 AAAAAAA SYS1 720896 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 106496 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 626688 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 827392 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 299008 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 110592 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 114688 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 86016 Apr -rwxr-x--- a-s- 1 AAAAAAA SYS1 106496 Apr
29 10:33 monitor_cs390
29 29 29 29 29 29 29 29 29
10:33 10:07 10:20 10:23 10:20 10:20 10:20 10:20 10:20
libITMNP.so libJLOG.so libcclog.dll libicmp.so libmsg23.so libtio.dll libtos.dll libtproc.dll libtthred.dll
Figure 6-2 Checking the APF security access attributes for monitor files

IBM Tivoli Monitoring for Network Performance monitors are started by the shell script /u/itmnp/V2R1M0/bin/itmnpMonitor.sh. The monitor output will be directed to the standard output of the shell process. You can direct the monitor messages to an output file as follows:
/usr/lpp/itmnp/V2R1M0/bin/itmnpMonitor.sh > /tmp/<out file> 2>&1 &
We created a procedure that uses BPXBATCH to run shell script itmnpMonitor.sh, which starts the monitor. This procedure enabled us to start the monitor from the system console, as shown in Figure 6-3, instead of having to start it from USS.
//ITMNPSTA PROC //ITNMPD EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT, // PARM='SH cd /usr/lpp/itmnp/V2R1M0/bin;./itmnpMonitor.sh' //STEPLIB DD DISP=SHR,DSN=DB8D8.SDSNLOAD // DD DISP=SHR,DSN=DB8D8.SDSNLOD2 //STDERR DD PATH='/var/ibm/tivoli/common/FNP/logs/itmnp.log', // PATHOPTS=(OWRONLY,OCREAT,OAPPEND), // PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP) //STDOUT DD PATH='/var/ibm/tivoli/common/FNP/logs/itmnp.log', // PATHOPTS=(OWRONLY,OCREAT,OAPPEND), // PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP) Figure 6-3 ITMNPSTA procedure to start the monitor
132
IBM Tivoli Monitoring for Network Performance should be stopped in the following sequence: 1. Stop the IBM Tivoli Monitoring for Network Performance monitor as follows: Issue the following command from the USS command line:
ps -ef | grep launcher
From this output, find the process ID and use it to issue the kill command below:
kill <process id>
where <process id> is the one belonging to the launcher. We created a procedure to stop the monitor from the system console, as shown in Figure 6-4.
//ITMNPSTO //ITNMPD // //STDERR // // //STDOUT // //
PROC EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT, PARM='SH /itmnp/V2R1M0/bin/itmnpStop.sh' DD PATH='/var/ibm/tivoli/common/FNP/logs/itmnp.log', PATHOPTS=(OWRONLY,OCREAT,OAPPEND), PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP) DD PATH='/var/ibm/tivoli/common/FNP/logs/itmnp.log', PATHOPTS=(OWRONLY,OCREAT,OAPPEND), PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP)
Figure 6-4 ITMNPSTO procedure to stop the monitor
The ITMNPSTO batch procedure executes the shell script, which is shown in Example 6-2. This shell script, which we created, finds the monitors process IDs and issues kill commands to stop the monitor.
Example 6-2 The itmnpsto.sh script to stop the monitor #!/bin/sh pid=`ps -ef | if [ $pid > 1 then kill -9 $pid fi pid=`ps -ef | if [ $pid > 1 then kill -9 $pid fi exit 0 grep launcher | grep -v grep | awk '{print $2}'` ]
grep monitor_cs390 | grep -v grep | awk '{print $2}'` ]
133
6.4 Sample monitor configuration

We started our monitors on WTSC62 and WTSC66 and created monitor definitions for these systems. From the IBM Tivoli Monitoring for Network Performance portfolio, we selected Configure Monitors z/OS Monitor and clicked Next on the Welcome page. On the Define Monitor Location page, we clicked Add Manually to display the Input Monitor Location page. Here we entered our sysplex name, system name, and fully qualified host name, as shown in Figure 6-5.
Figure 6-5 Monitor location page
After clicking OK, we were returned to the Monitor Location page, as shown in Figure 6-6 on page 135. We then clicked Autoconfigure. By using the autoconfigure option, we opted to use the IBM default values for our collection schedule, which uses default values for subsequent steps and navigates directly to the Save Monitor definition page.
134
Figure 6-6 Monitor location autoconfigure screen
After saving and naming our monitor definition, we were returned to the Summary page, as shown in Figure 6-7 on page 136.
135
Figure 6-7 Monitor Autoconfigure summary
On the Summary page, we clicked Finish to save the monitor definition and close the wizard. To get the monitor to use this configuration, we deployed the configuration by clicking on Manage Monitors Manage Monitor Configuration, clicking on the system name, and then on Deploy Configuration, as shown in Figure 6-8 on page 137.
136
Figure 6-8 Deploying monitor configuration
Multiple monitor definitions can be created. One would typically create definitions for each system, a group of systems, or for resources that are shared by multiple systems. Monitor definitions could include exceptions for public holidays and scheduled outages. Refer to the IBM Tivoli Monitoring for Network Performance Administrator Guide, SC31-6364 for a description of how to create z/OS Monitor, SNMP Monitor, and Availability and Response Time Monitor definitions.
6.5 Monitoring best practices

In the z/OS Communication Server, there is plenty of data that is really useful for problem determination and troubleshooting. The major resources being used by the z/OS Communication Server are CPU, network interfaces, network connections, or applications and storage. The CPU consumption can be monitored using other software, such as RMF, for example. The other resources can be monitored by IBM Tivoli Monitoring for Network Performance.
137
The data collected can be used to generate system standards or system profiles. It will define the system behavior in terms of network, such as what applications are up, the total number of connections, what network interfaces are being used, and their utilization. Therefore, those profiles can be used to set thresholds or events whenever something unusual happens. There are some other tools that can be used to generate profiles, such as the Policy Agent that comes with z/OS Communication Server. It is possible to configure it based on the data collected by IBM Tivoli Monitoring for Network Performance. Refer to the following documentation on how to configure the Policy Agent feature: z/OS V1R6 Communications Server: IP Configuration Reference, SC31-8776 z/OS V1R6 Communications Server: IP Configuration Guide, SC31-8775 Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 6: Policy and Network Management, SG24-6839 This section discusses some ideas and suggestions on how to configure the monitors for IBM Tivoli Monitoring for Network Performance. The discussion is divided into: 6.5.1, Monitored metrics on page 138 6.5.2, Monitoring configuration design on page 140 6.5.3, Monitoring information on page 142 6.5.4, Monitoring storage usage on page 153 6.5.5, Monitoring network interfaces on page 157
6.5.1 Monitored metrics

In IBM Tivoli Monitoring for Network Performance, there are three types of monitors: z/OS Communication Server monitor. This monitor uses the Communication Server API and SNMP collection to gather performance metrics for the z/OS Communication Server TCP/IP function. Availability and response time monitor. This monitor uses the ping command to collect response time and availability of a remote host from the monitor. SNMP monitor. This monitor collects Management Information Base (MIB) variables from remote nodes and stores them in the database. Each monitoring configuration consists of the following information: The monitor host that will perform this monitoring Monitor type
138
Monitoring target, such as TCP/IP stack (for z/OS monitor) or IP address list (for SNMP and ICMP monitor) Data or metric to collect, depending on the monitor types Threshold and rearm values Schedules on which collection will be performed The following lists the information that is collected by each type: z/OS Communication Server monitor TCP stack performance TCP/IP application performance TCP connection performance UDP stack performance IP stack performance FTP performance TN3270 performance Interface performance data TCP/IP memory usage HPR and EE performance data CSM buffer data OSA performance data Availability and Response Time monitor Response time information Availability statistic SNMP monitor Cisco router performance statistics Cisco switch performance statistics Interface statistics RMON Ethernet performance
139
6.5.2 Monitoring configuration design

In our design for monitoring, we consider the following aspects: Schedules of the monitors. These schedules must be consistent with each other and present the least number of interruptions in the production environment. The threshold and rearm setting and event forwarding need to be specified wisely to minimize useless alerts, but still forward all the necessary events. Data selection on what information to collect for which node. Let us consider each aspect one by one.
Schedule consideration
There are several recommendations that we have after analyzing the collected data: Use a single interval value or a multiplication of that interval for all monitors. The use of the same interval allows consistent reports over all schedules. Usage of the same interval for different monitors also allows cross check of data for different monitors and different hosts. Splitting the schedule for prime-time and off-hours may not be really necessary. We prefer to collect a standard amount of information continuously than a different set of information at discrete times. This is easier to analyze. Emergency schedules for problem determination may be prepared to be activated as needed. Remember that all this information is collected into the same database and the same tables. You may need to analyze the data using SQL commands instead of the Web application to get meaningful insight. Note: Remember that FTP and TN3270 are not collected based on the interval but as the session completed.
Threshold and rearm value

The consideration for putting in the threshold and rearm values are: Put as much threshold as possible, and supply the rearm value only if it is meaningful. Change these thresholds values as you learn more on the performance of your systems. You may need to create a different threshold set for each z/OS TCP/IP stack, different SNMP target, or different IP addresses. Each monitor will typically be unique. You can start them by using the IBM supplied value, if available.
140
Only forward events that are used by automation to respond; do not flood the automation subsystems with unused events.
Collection data
We recommend you define the collection data or at least make the data collection plan before you actually define it in the IBM Tivoli Monitoring for Network Performance Web application. This will identify the necessary resources and possible holes in the collected data. Different monitoring types require different considerations: z/OS Communication Server monitoring For each z/OS TCP/IP stack, you need to determine whether you need to monitor that stack (for example, test TCP/IP processes). Consider whether you need to collect FTP and TN3270 data. This information is collected as it becomes available. This may introduce unnecessary traffic on the database. Open System Adapter (OSA) requires that you run the OSA Service Facility (OSASF) started task. Set this up before activating the OSA monitoring. Consider whether you have High Performance Routing (HPR) needs. Consider whether you have Enterprise Extender (EE) monitoring needs. Availability and response time data You may want to collect response time and availability information for remote subnets that need a connection to the z/OS. Use a node or address that is expected to be always available, such as a router interface or a switch control port. You may have one or more samples from a remote subnet. It is not very useful to collect this information on a local subnet unless you need the data for Service Level measurement. You may want to always forward unavailability events and bad performance events for monitoring to switches and routers. Other nodes to monitor are distributed servers that connect to the mainframe. This collection uses the ping protocol; be selective on choosing the node to monitor. SNMP performance data Typically, you collect this information for routers and switches that are critical for the availability of the network.
141
As the SNMP performance data is a polling collection, it may generate a large amount of data a single time, and may obscure other performance metrics when collected at the same time. You may want to have a longer interval for SNMP collection. For example, if you collect z/OS communication server and ICMP data every 15 minutes, you may choose to collect SNMP performance every hour.
6.5.3 Monitoring information

The following sections display and discuss some sample monitoring results. The discussion is divided into the following topics: Monitoring TCP/IP stack on page 142 Monitoring applications on page 148 Monitoring storage usage on page 153 Monitoring network interfaces on page 157
Monitoring TCP/IP stack

The TCP/IP stack is the major component of a z/OS image TCP/IP network. It is responsible for providing the IP, TCP, and UDP services to all applications that require those services, such as TN3270 server, CICS, FTP client and server, DB2, IMS, WebSphere, and MQSeries. The IBM Tivoli Monitoring for Network Performance has three different kinds of information on the stack monitoring: TCP protocol UPD protocol IP protocol The TCP protocol set of panels has information on TCP connections, giving a more general view of the stack. Figure 6-9 on page 143 can be used to check the number of active connections, the number of accepted connections, the number of connections dropped, and the connection rate in a specific period of time. Having all of those samples for a specific shift, such as 9:00 AM to 12:00 AM, during a month, only on week days, or from Monday to Friday, it is possible to create a profile for those stacks, for example, based on the number of connections.
142
Figure 6-9 TCP stack availability and response
After having a profile for a specific stack, for example, the average number of connections for a specific stack is 5000 and the connection rate is 20 per second. This information allows the creation of a threshold to generate alerts when there are more than 6000 connections and more than 30 connections being accepted at a given time. This means that something unusual happened in this period and further investigation can be performed to find out what is going on (see Monitoring applications on page 148 for an example). Figure 6-10 on page 144 showed how to define this threshold (select Monitor Configuration Set Thresholds TCP Stack Performance Data Web Application).
143
Figure 6-10 Stack availability and response setting threshold
Those monitor configurations can be dynamically changed at anytime. Actually, sometimes there will be some changes on the application or in the system behavior that will require changes on the thresholds to avoid generating some events that are not related to problems. There is some information about the traffic flow regarding the TCP applications running on a z/OS system. Figure 6-11 on page 145 shows, for example, the amount of segments being transmitted and received on a specific system.
144
Figure 6-11 TCP stack throughput and traffic
For the UDP application, there are two pages showing the same information from the UPD protocol perspective. One of the major users of the UDP protocol will be the SNMP daemon. In fact, to implement the IBM Tivoli Monitoring for Network Performance, it is necessary to have the OSMPD daemon running on the z/OS system. There are two pages with UDP protocol information: The whole TCP/IP stack UDP traffic overview, as shown on Figure 6-12 on page 146.
145
Figure 6-12 UDP stack throughput and traffic
Per the UDP application overview, as shown on Figure 6-13 on page 147. The columns are reordered to fit on screen; this is not the default view and it is showing three different OSMNPD applications running on three different z/OS images. A filter was applied to show only the job name that contains the smnp string.
146
Figure 6-13 UDP endpoint throughput and traffic
For the IP protocol, the IBM Tivoli Monitoring for Network Performance has only one panel, which contains information about the manipulated datagrams for each one of the systems being monitored. Figure 6-14 on page 148 shows an example of this situation.
147
Figure 6-14 IP stack throughput and traffic
Monitoring applications
Most of the applications running on a z/OS system support the TCP/IP protocol on some level. Each one has its own application specific protocol built under the TCP/IP protocol. For example, for terminal emulation, we have the UNIX Telnet protocol and the TN3270 protocol. For file transfer, we have the FTP protocol, and for Web applications, we have the HTTP protocol. Those are well-known TCP/IP application protocols, but there are many others. Each one of these protocols is identified by the TCP/IP stack by ports. Each port will be listening, or accepting connections, to a specific application protocol, so there will be a specific address space listening on a specific port. It is possible to have the same address space listening on more than one port at the same time, like the HTTP Server for normal and SSL connections. The TN3270 server can have 256 different ports at the same time for normal and SSL connections. The IBM Tivoli Monitoring for Network Performance cannot understand what is happening within the application protocol, but it can monitor the application
148
behavior in terms of the number of active connections, the number of connections accepted, bytes sent and received, and so on. For the TN3270 and FTP applications there are a special set of pages show specific information for those two application protocols. For the FTP applications, there are three different pages: The active FTP sessions page shows, on a specific time, all server and client sessions from the z/OS images. As an example, see Figure 6-15.
Figure 6-15 FTP sessions
The z/OS FTP client transfer records page shows only the information about client sessions started from a z/OS system to a remote FTP server. The z/OS FTP server transfer records page show remote clients connections to a z/OS local FTP server. This page shows complete information about file transmissions that occurred on a specific period and can be used, for example, to monitor the activity of specific users. See Figure 6-16 on page 150.
149
Figure 6-16 FTP server transfer records
The TN3270 application is one of the most popular TCP applications running on z/OS systems. There are three pages showing the TN3270 information: The TN3270 server session availability page shows all TN3270 remote clients connected to a TN3270 server running on a z/OS image. Each one of the records will show a complete TN2370 session. As an example, it will show the start time, end time, local and remote IP addresses and ports, and the number of transmitted bytes in a specific TN3270 session. The TN3270 sliding-window average response time page shows the response time for the IP and SNA portions of a TN3270 session. This will only work on z/S V1.5 systems or later when selecting the option to collect the TN3270 performance data on monitor definition. The TN3270 response time counts by time bucket page shows the number of times a response times fits in each response time interval (the buckets). This only works on z/OS V1.5 systems or later when selecting the option to collect the TN3270 performance data on monitor definition.
150
Attention: In z/OS V1.5, the Communication Server TN3270 server has two new configuration statements to define monitor options. These new statements are MONITORGROUP and MONITORMAP. Refer to z/OS V1R6 Communications Server: IP Configuration Reference, SC31-8776 for further information on how to define those parameters and collect more detailed performance data on a TN3270 Server connection. As we are running the monitors on z/OS V1.4, we do not have samples to show. For the other applications, IBM Tivoli Monitoring for Network Performance has features that can help monitor them. We selected a specific address space, in this case, the DB2 TCP/IP application. By using filters, we can have some information about what is going on with it. The shows the number of active connections, the maximum number of connections, the date and time for this number, and the idle time since the last connection started. Using this information, it is possible to show that the DB2 address space was listening and accepting connections at a given time. See Figure 6-17 for an example.
Figure 6-17 Other application availability and response
151
It is possible to do the same thing on the connection availability and response page, which shows the applications response time. See Figure 6-18.
Figure 6-18 Other connection availability and response
The connection throughput and traffic page shows the traffic rate for the same application. See Figure 6-19 on page 153.
152
Figure 6-19 Other connection throughput and traffic
Using this method, it is possible to monitor all TCP/IP applications.
6.5.4 Monitoring storage usage

To run the applications, the TCP/IP stack has to create control blocks to manage all the connections and to do the network I/O. Those control blocks are used to manage all the connections and they will be built under the TCP/IP address space or in the common storage area, for example, ECSA. There are some guidelines on how to estimate and how to do capacity planning to prevent storage shortages on a z/OS system. For each application, there is a different way to estimate the storage requirement for: The TCP/IP address space The application address space Common Storage Using IBM Tivoli Monitoring for Network Performance, it is possible to monitor: The storage consumption in the private area and the ECSA common storage of the TCP/IP address space.
153
The storage allocated by the Communications Storage Manager (CSM). For the application address space, look at the application specific information to monitor. The only application that allocates storage in the TCP/IP stack address space is the TN3270 server. Since z/OS Version 1 Release 6, it is possible to run the TN3270 server in a separate address space and the memory will be allocated on it, outside of the TCP/IP stack address space. The CSM is a Communication Server component, share by the TCP/IP and VTAM applications, that permits sharing data between their users, like the FTP server and other applications, avoiding the data movement between them and reducing the overhead. Refer to the z/OS Communications Server CSM Guide Version 1 Release 2, SC31-8808 for more information. Another important task to perform when using TCP/IP applications under z/OS is to do storage capacity planning. Chapter 7, Performance and tuning, of Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 5: Availability, Scalability, and Performance, SG24-6517 has information about how to do a storage capacity planning on some applications. It is important to monitor the storage consumption to avoid unexpected TCP/IP stack restart or even initial program loads due to memory leak problems. The storage consumption for a given workload should be stable. If the storage consumption is being increased without any workload growth, like the number of connections, especially TN3270 connections, there is probably something wrong, and you should contact the IBM Support Center to check on the latest z/OS Communication Server service maintenance. Here are some examples from IBM Tivoli Monitoring for Network Performance Web Application pages on how to monitor the storage consumption. In Figure 6-20 on page 155, it is possible to gather information about how much memory is being allocated by the TCP/IP address space. It is possible to match this information against applications running on the z/OS image and create a correlation between them at a given time. The same can be done with the CSM Storage pages.
154
Figure 6-20 TCP/IP storage statistics
In Figure 6-21 on page 156, it is possible to monitor how much CSM storage is being allocated by the CSM, the maximum CSM storage allocation allowed on a z/OS image, and the percentage of CSA that is being allocated by the CSM.
155
Figure 6-21 CSM storage summary
If more detailed information is needed, like how much storage is being allocated per pool type, refer to the CM storage monitoring page. See Figure 6-22 on page 157.
156
Figure 6-22 CSM storage monitoring
6.5.5 Monitoring network interfaces

The network interfaces are used as paths to communicate with the network. A single z/OS image can have different types of interfaces, such as OSA express cards, XCF communication links, and channel attached interfaces, such as Cisco and IBM routers. The most used network interfaces are the OSA adapters. OSA adapters have two different configurations: the QDIO mode and the non-QDIO mode or LCS adapters. To learn more about OSA adapters, refer to the following publications: OSA-Express Customers Guide and Reference, SA22-7935 and SA22-7476 OSA-Express Implementation Guide, SG24-5948 OSA-2 Implementation Guide (Update), SG24-4770 The network interfaces have to be monitored to prevent network congestion and to ensure that the network definitions are being followed. One of the ways to verify the network routes and load balancing options is by checking the traffic flow for inbound and outbound directions.
157
To learn more about network interfaces, look at the following documentation: Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 4: Connectivity and Routing, SG24-6516 z/OS V1R6 Communications Server: IP Configuration Guide, SC31-8775 z/OS V1R6 Communications Server: IP Configuration Reference, SC31-8776 The status page shows the interfaces on all systems. Figure 6-23 shows the interfaces capabilities. There is a filter applied to this page that selects only the interfaces that have osa on their names.
Figure 6-23 Interface status
In Figure 6-24 on page 159, on the same status page, it is possible, as an example, to see the maximum MTU supported by the interfaces.
158
Figure 6-24 Interface status (continued)
In Figure 6-25 on page 160, it is possible to monitor the interface statistics, such as the traffic rate. It is possible to see, for example, if there are two interfaces connected to the same lan and the TCP/IP profile is set to use multipath; transmitted octets for both interfaces have to be similar.
159
Figure 6-25 Interface unicast performance metrics
For the multicast and broadcast traffic, there is a specific page, Interface Multicast/Broadcast Performance Metric, which shows information about these protocols. If the OSPF routing protocol is being used, it will generate some data for the multicast traffic.
160
Chapter 7.
Discovery and alert interfaces

This chapter discusses the interfaces for IBM Tivoli Monitoring for Network Performance for TCP/IP resource discovery and the alerting interface. TCP/IP resource discovery is assisted by the IBM Tivoli NetView Integrated TCP/IP Services Component (ITSC), while the alerting interface uses IBM Tivoli Enterprise Console and IBM Tivoli NetView for z/OS. The discussion is divided into the following topics: 7.1, NetView Integrated TCP/IP services component on page 162 7.2, Event integration with IBM Tivoli Enterprise Console on page 164 7.3, Event integration with IBM Tivoli NetView for z/OS on page 175
161
7.1 NetView Integrated TCP/IP services component

This section discusses the interface from NetView ITSC to IBM Tivoli Monitoring for Network Performance. The discussion is divided into the following topics: 7.1.1, Concepts on page 162 7.1.2, Configure NetView Integrated TCP/IP Services Component on page 162 7.1.3, Verification on page 163
7.1.1 Concepts
The discovery functions use the IBM Tivoli NetView product. IBM Tivoli NetView is also called the Integrated TCP/IP Solution Component (ITSC). IBM Tivoli Monitoring for Network Performance requires IBM Tivoli NetView Version 7.1.4, as it has the ability to detect a z/OS node. The discovery function adds a new daemon for IBM Tivoli NetView, which is called nvexportd. This process is a Java daemon started by the spmsur program. The nvexportd daemon connects to the itmnpItsc Web application servlets and uploads subnets, nodes, and smartsets information to IBM Tivoli Monitoring for Network Performance. This daemon can be started using NetView standard commands ovstart nvexportd and ovstop nvexportd. Its status can be queried using the command ovstatus nvexportd. The discovery function uses NetViews netmon discovery, including the preparation for a seed file. We recommend including the z/OS systems in the netmon seed file for quick discovery. As the communication from NetView ITSC to z/OS uses SNMP, which is UDP based, this typically cannot get through a firewall. We recommend having a separate NetView machine on each local subnet that has z/OS monitored.
7.1.2 Configure NetView Integrated TCP/IP Services Component

We installed IBM Tivoli NetView on a Windows server and installed the nvexportd component of the IBM Tivoli Monitoring for Network Performance on a Windows server. We follow the instructions in Chapter 8, Setting Up the NetView Integrated TCP/IP Services Component, of IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363.
162
Example 7-1 shows that the NetView Integrated TCP/IP Services Component is up and running.
Example 7-1 Status of nvexportd C:\usr\ov\conf>ovstatus nevexportd object manager name: nvexportd behavior: OVs_WELL_BEHAVED state: RUNNING PID: 2232 last message: Initialization complete. exit status: Done
7.1.3 Verification
We checked that the NetView Integrated TCP/IP Services Component sent the network resources to the Web application by opening a browser to the following URL:
https://jakarta:9454/itmnpitsc/BrowseContents
Figure 7-1 on page 164 shows that the NetView Integrated TCP/IP Services Component sent the network resources to WebSphere Application Server as XML-formatted data.
Chapter 7. Discovery and alert interfaces
163
Figure 7-1 Display of IP-resources from WebSphere Application Server
7.2 Event integration with IBM Tivoli Enterprise Console

IBM Tivoli Monitoring for Network Performance monitors can be configured to generate events when specific thresholds are crossed. Events can be received by any application that is designed to receive Event Integration Facility (EIF) events, such as IBM Tivoli Enterprise Console (TEC). An overview of IBM Tivoli Enterprise Console is provided in IBM Tivoli Enterprise Console Version 3.9 on page 235. The discussion in this section is divided into the following topics: 7.2.1, Customizing TEC rulebase on page 165 7.2.2, Configuring event forwarding on page 168 7.2.3, Sample event automation program on page 173
164
7.2.1 Customizing TEC rulebase

IBM Tivoli Enterprise Console is an automation application based on a rulebase. A rulebase contains a set of baroc classes that defines the event formats and structure and a set of rulesets that defines the processing for those events. In order for IBM Tivoli Enterprise Console to understand the events sent from IBM Tivoli Monitoring for Network Performance, it needs the baroc class definition in its rulebase. This baroc class is supplied by the file itmnp.baroc from the IBM Tivoli Monitoring for Network Performance CD-ROM. This file needs to be copied to a local directory of the IBM Tivoli Enterprise Console server. Customization of the rulebase can be performed using the command line or graphical interface with Tivoli Desktop. The command line processing is explained in detail in IBM Tivoli Monitor for Network Performance: Planning, Installing, and Configuring, SC31-6363. We assume that Tivoli Framework and IBM Tivoli Enterprise Console are installed and running.
Command line processing

We install the IBM Tivoli Monitoring for Network Performance classes to a new rulebase as follows: 1. We create a new rulebase called itmnp using the command wrb -crtrb itmnp -d C:/Tivoli/bin/w32-ix86/TME/TEC/itmnp_rb. 2. We import the itmnp.baroc into the new rulebase using the command wrb -imprbclass C:/itmnp.baroc itmnp. This command copies the itmnp.baroc file into the TEC_CLASSES directory C:/Tivoli/bin/w32-ix86/TME/TEC/itmnp_rb/TEC_CLASSES. 3. We compiled the itmnp.baroc using the command wrb -comprules itmnp. Example 7-2 shows the output of compiled rule. This will create the load rule base for TEC.
Example 7-2 Output of the compile rule base bash$ wrb -comprules itmnp Compiling itmnp Loading Classes Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\root.baroc Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\tec.baroc Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\itmnp.baroc Loading Rule Sets Loading Rule Packs Loading RB Targets Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_RULES\EventServer Translating Rules Compiling Rules
165
Building RB target EventServer Compiling C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\.rbtargets\EventServer\TEC_RUL ES\.load_rules.pro Compilation Complete.
4. We loaded the rule base using the wrb -loadrb itmnp command, as shown in Example 7-3.
Example 7-3 Display of the load rule base. bash$ wrb -loadrb itmnp Loading Classes Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\root.baroc Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\tec.baroc Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_CLASSES\itmnp.baroc Loading Rule Sets Loading Rule Packs Loading RB Targets Parsing C:\Tivoli\bin\w32-ix86\TME\TEC\itmnp_rb\TEC_RULES\EventServer
5. We stopped the TEC using the wstopesvr command. 6. We restarted the TEC with the wstartesvr command. The following message indicates the TEC server has started:
The Tivoli Enterprise Console Server is running.
Tivoli Desktop interface

The same processing can be performed using the Tivoli Desktop. 1. From the Tivoli Desktop, find the Event Server icon. Double click to open the Event Server and get the list of rule bases, you will see at least the Default rule base. 2. From the menu, select Create Rule Base, as shown in Figure 7-2 on page 167. In the New rule base dialog, specify the name and directory of the new rule base. Click on Create & Close.
166
Figure 7-2 Create a new rule base
3. Right-click on the created rule base and select Import, as shown in Figure 7-3. Check the import class and input the path of the itmnp.baroc file. Click on Import & Close.
Figure 7-3 Importing a baroc file
4. Right-click on the itmnp rulebase and select Compile. The compilation window is shown in Figure 7-4 on page 168.
167
Figure 7-4 Compile rulebase window
5. Right-click on the itmnp rulebase and select Load & Close. The load window is shown in Figure 7-5.
Figure 7-5 Load rulebase window
6. Close the rulebase window. 7. Right-click on the Event Server icon and select Shut down. Wait until the red arrow disappears and then right-click again and select Start Up. When the red arrow reappears as a solid object, the process has completed.
7.2.2 Configuring event forwarding

Now that the IBM Tivoli Enterprise Console has been configured to receive IBM Tivoli Monitoring for Network Performance events, we need to tell the monitors to start sending the events to our IBM Tivoli Enterprise Console server. 1. First, we need to have a configuration that has the forwarding request. We defined the FTP threshold event and checked the generate event field in the configure z/OS monitor, as shown in Figure 7-6 on page 169.
168
Figure 7-6 Setup to generate an event
2. We saved and deployed the monitor configuration to the monitor. 3. We defined the IBM Tivoli Enterprise Console IP address and port in the IBM Tivoli Monitoring for Network Performance by selecting Set Run-time Preferences Define Event Receivers. Click on the Add button, as shown in Figure 7-7 on page 170. The default port number for IBM Tivoli Enterprise Console is 5529.
169
Figure 7-7 Defining the event receiver
4. We simulated the FTP server transfer records exceeding the set threshold to generate the event. Figure 7-8 on page 171 shows the event generated on IBM Tivoli Monitoring for Network Performance.
170
Figure 7-8 Threshold crossed is marked with a red X circle
5. Figure 7-9 on page 172 shows the event received in the TEC event.
171
Figure 7-9 Display of TEC event view
The detailed event structure is shown in Figure 7-10 on page 173.
172
Figure 7-10 Detailed event structure
7.2.3 Sample event automation program

In this section, we provide a sample ruleset and program to process the event. One problem with the event from IBM Tivoli Monitoring for Network Performance view is that all events use ITMNP_ThresholdExceeded or ITMNP_ThresholdRearmed. The important information about the measurement is contained in the following slots: Hostname: The host name where the monitor that sends the event executes field_name: The actual threshold that get tripped field_value: The threshold value that causes the event The sample ruleset that we use is shown in Example 7-4 on page 174.
173
Example 7-4 Sample itmnp.rls file rule: itmnp_threshold: ( event: _event of_class within ['ITMNP_ThresholdExceeded'] where [ ], reception_action: invoke_Automation: ( exec_program(_event, 'scripts/itmnp_tec.sh', '', [], 'YES'), commit_set ) ). rule: itmnp_rearmed: ( event: _event of_class within ['ITMNP_ThresholdExceeded'] where [ field_name: _field_name, monitor_host: _monitor_host ], reception_action: invoke_Automation: ( exec_program(_event, 'scripts/itmnp_tec.sh', '', [], 'YES'), commit_set ) reception_action: close_Threshold_event: ( first_instance(event: _original_event of_class 'ITMNP_ThresholdExceeded' where [ field_name: equals _field_name, monitor_host: equals _monitor_host ] ), set_event_status(_original_event, 'CLOSED'), commit_set ) ).
This rule selects all IBM Tivoli Monitoring for Network Performance events and executes the itmnp_tec.sh program. The itmnp_tec.sh tool needs to parse and validate the event information and act appropriately. Our sample script just parses the event and writes to an output file. A real program needs to perform additional functions, such as sending e-mail or a page to an operator. Our itmnp_tec.sh is shown in Example 7-5 on page 175.
174
Example 7-5 The itmnp_tec.sh #!/bin/sh LOGFILE=/tmp/itmnpevents.log echo $EVENT_CLASS $date_reception $monitor_host $field_name $field_value >> $LOGFILE # # # # # if you want to send this to TBSM based on the measurement assuming all event goes to ITMNP 2.1 object class $TEC_BIN_DIR/../../TDS/EventService/ihstttec -b "ITMNP;2.1" \ -i "$monitor_host" -p "$field_name" -s "$severity" -o 20 \ -1 "$field_value" -t "EXCEPTION" -q "$origin" -m "$msg"
####################################### # event specific processing goes here if [ "$EVENT_CLASS" = "ITMNP_ThresholdExceeded" ]; then # Here are processing specific to Threshold tripped event $TEC_BIN_DIR/bin/TEC_Send_Mail.sh "ITMNP-TEC interface" tec_itmnp "$msg" else # Here are processing specific to Rearmed events fi
7.3 Event integration with IBM Tivoli NetView for z/OS

To alert an operator about performance issues and threshold value violations, rearm condition events have to be set in IBM Tivoli Monitoring for Network Performance. These generated events can then be sent to IBM Tivoli NetView for z/OS. IBM Tivoli NetView for z/OS must be configured to receive these events using the event-to-alert conversion facility of the Event Automation Service (EAS). This discussion is similar to 7.2, Event integration with IBM Tivoli Enterprise Console on page 164. It consists of the following topics: 7.3.1, Setting up Event Automation Service on page 176 7.3.2, Defining threshold and event generation on page 177 7.3.3, Automating NetView alert on page 181
175
7.3.1 Setting up Event Automation Service

Event Automation Service provides an interface between IBM Tivoli NetView for z/OS and IBM Tivoli Enterprise Console. It can send NMVT alerts as TEC events and receive TEC events as NMVT alerts. Here we use the function that receives a TEC event and converts it to a Generic NMVT alert (GENALERT). The following steps set the Event Automation Service. For more information, refer to Tivoli NetView for z/OS, Installation: Configuring Additional Components Version 5 Release 1, SC31-8874. 1. We copy the Event Automation Service started task procedure, IHSAEVNT, from NETVIEW.SCNMUXMS to our PROCLIB. 2. We customize the IHSAEVNT member and change the reference to NetView datasets in our environment, SQ1=NETVIEW and UQ1=NETVUSER.SC61N. 3. We set the STEPLIB dataset NETVIEW.SCNMUXLK to be APF authorized by using the SETPROG APF,ADD,DSN=NETVIEW.SCNMUXLK,VOL=****** command. 4. We change the event receiver configuration file in dataset NETVIEW.SCNMUXCL(IHSAECFG) to include the statement PortNumber=9162. If the port number is not specified or the value is zero, then a port number will be dynamically assigned. A static port number is required for the event forwarding from IBM Tivoli Monitoring for Network Performance. 5. We also ensure that the event receiver task is started with the initialization of the EAS. The initialization member is NETVIEW.SCNMUXCL(IHSAINIT). You need to ensure that the line NOSTART TASK=EVENTRCV is commented out. 6. We start the EAS using the command MVS S IHSAEVNT from NetView session. We then check the PPI registration from the EAS using the DISPPI command. It should have results similar to Example 7-6.
Example 7-6 DISPPI result * SC61N ' SC61N DWO948I DWO949I DWO950I DWO951I DWO951I DWO951I DWO951I DWO951I DWO968I DISPPI RECEIVER RECEIVER BUFFER IDENTITY STATUS LIMIT -------- -------- ---------NETVALRT ACTIVE 1000 DSIMCAT ACTIVE 25 ISTMTRCV ACTIVE 500 SC61NHTM ACTIVE 1000 NETVRCV ACTIVE 500 END OF DISPLAY QUEUED TOTAL BUFFERS BUFFERS ---------- ---------0 0 0 0 0 1 0 0 0 2 STORAGE ALLOCATED ---------0 0 0 0 0 RCVR ASID ---0020 0020 001C 0020 0020
176
7.3.2 Defining threshold and event generation

The process of determining a meaningful set of threshold and rearm values for your environment will most likely be an interactive process. In our environment, we set IBM Tivoli Monitoring for Network Performance to forward alerts to IBM Tivoli NetView for z/OS. The process to set up forwarding and threshold is similar to 7.2.2, Configuring event forwarding on page 168. This event was forwarded to IBM Tivoli NetView for z/OS EAS and displayed in the hardware monitor (NPDA) events display. Tip: If it is indicated that events have been generated and you do not see the event in NPDA, you may have active NPDA filters. Use the following REXX program to clear the event filters:
/* REXX CLIST to Delete Default Alert Filter Settings */ 'NPDA SRF AREC DELETE E HELD TREF CTRL' 'NPDA SRF AREC DELETE E PERM TREF CTRL' 'NPDA SRF AREC DELETE E PERF TREF CTRL' 'NPDA SRF AREC DELETE E HELD TREF LCTL' 'NPDA SRF AREC DELETE E PERM TREF LCTL' 'NPDA SRF AREC DELETE E PERF TREF LCTL' 'NPDA SRF AREC DELETE E HELD' 'NPDA SRF AREC DELETE E PERM' 'NPDA SRF AREC DELETE E USER' 'NPDA SRF AREC DELETE E NTFY' 'NPDA SRF AREC DELETE E INST' 'NPDA SRF AREC DELETE E SCUR' 'NPDA SRF AREC DELETE E UNKN' 'NPDA SRF AREC DELETE E PERF' 'NPDA SRF AREC DELETE TREF CPU' 'NPDA SRF AREC PASS DEFAULT' exit
Figure 7-11 on page 178 shows the event history page in NetView.
177
N E T V I E W NPDA-41A SC61N
SESSION DOMAIN: SC61N VBUDI * MOST RECENT EVENTS *
06/15/04 18:34:22 PAGE 1 OF 1
WTSC61.I 9.12.4.3 ITMNP +--------+ +--------+ +--------+ DOMAIN | PWS |---| NTID |---| APPL | +--------+ +--------+ +--------+ SEL# DATE/TIME EVENT DESCRIPTION:PROBABLE CAUSE ( 1) 06/01 00:39 wtsc61.itso.ibm.com: Thre:severity=HARMLESS; ( 2) 05/31 21:57 wtsc61.itso.ibm.com: Thre:severity=WARNING; ( 3) 05/31 09:51 wtsc61.itso.ibm.com: Thre:severity=HARMLESS; ( 4) 05/31 08:36 wtsc61.itso.ibm.com: Thre:severity=WARNING; ( 5) 05/30 12:22 wtsc61.itso.ibm.com: Thre:severity=HARMLESS; ( 6) 05/30 11:31 wtsc61.itso.ibm.com: Thre:severity=WARNING; ( 7) 05/29 17:02 wtsc61.itso.ibm.com: Thre:severity=HARMLESS; ( 8) 05/29 15:54 wtsc61.itso.ibm.com: Thre:severity=WARNING; ( 9) 05/29 06:52 wtsc61.itso.ibm.com: Thre:severity=HARMLESS; (10) 05/29 06:12 wtsc61.itso.ibm.com: Thre:severity=WARNING; ENTER ST (STAT), SEL# (ACTION), OR SEL# PLUS D (EVENT DETAIL) ??? CMD==> Figure 7-11 Event history for SC61
ETYP TEMP IMPD TEMP IMPD TEMP IMPD TEMP IMPD TEMP IMPD
Selecting one of these WARNING events gives the threshold exceeded event, as shown in Figure 7-12 on page 179.
178
N E T V I E W NPDA-43S SC61N DOMAIN
SESSION DOMAIN: SC61N VBUDI * EVENT DETAIL *
06/15/04 18:37:56 PAGE 1 OF 2
WTSC61.I 9.12.4.3 ITMNP +--------+ +--------+ +--------+ | PWS |---| NTID |---| APPL | +--------+ +--------+ +--------+
HIERARCHY NAMES LIST: PWS wtsc61.itso.ibm.com NTID 9.12.4.32 APPL ITMNP DATE/TIME: RECORDED - 05/31 21:57 EVENT TYPE: IMPENDING PROBLEM DESCRIPTION: wtsc61.itso.ibm.com: Threshold tripped [ PROBABLE CAUSES: severity=WARNING; ORIGINAL T/EC EVENT: ITMNP_ThresholdExceeded; monitor_host=wtsc61.itso.ibm.com; source=ITMNP; msg=wtsc61.itso.ibm.com: Threshold tripped [TCP_CONN_AVL_MSMT.PERC_SEG_RETRANS = 3.252033 (> 3)]; field_name=TCP_CONN_AVL_MSMT.PERC_SEG_RETRANS; msmt_details=Category=null;LOCAL_NODE_IP='9.12.4.33';REMOTE_PORT=3628;RETRA NS_RT=0.001674;RESP_TM_VAR=38;RETRANS_SEGS=4;START_TM='2004-05-27-03.29.46. 000000';REMOTE_NODE_IP='9.155.177.145';REM_WINDOW_CNT=0;APP_JOB_NM='DFSKERN ';LAST_ACTIV msmt_id=c8b49449090c0420000000fcd9417732; timestamp=2004-05-30 23:43:42.0; adapter_host=wtsc61; hostname=wtsc61.itso.ibm.com;
ENTER A (ACTION) OR DM (DETAIL MENU) ??? CMD==> Figure 7-12 Threshold exceeded event
The corresponding rearm event is shown in Figure 7-13 on page 180.
179
N E T V I E W NPDA-43S SC61N DOMAIN
SESSION DOMAIN: SC61N VBUDI * EVENT DETAIL *
06/15/04 18:39:53 PAGE 1 OF 2
WTSC61.I 9.12.4.3 ITMNP +--------+ +--------+ +--------+ | PWS |---| NTID |---| APPL | +--------+ +--------+ +--------+
HIERARCHY NAMES LIST: PWS wtsc61.itso.ibm.com NTID 9.12.4.32 APPL ITMNP DATE/TIME: RECORDED - 06/01 00:39 EVENT TYPE: TEMPORARY DESCRIPTION: wtsc61.itso.ibm.com: Threshold rearmed [ PROBABLE CAUSES: severity=HARMLESS; ORIGINAL T/EC EVENT: ITMNP_ThresholdRearmed; monitor_host=wtsc61.itso.ibm.com; source=ITMNP; msg=wtsc61.itso.ibm.com: Threshold rearmed [TCP_CONN_AVL_MSMT.PERC_SEG_RETRANS = 0.000000 (<= 1)]; field_name=TCP_CONN_AVL_MSMT.PERC_SEG_RETRANS; msmt_details=Category=null;LOCAL_NODE_IP='9.12.4.33';REMOTE_PORT=3628;RETRA NS_RT=0.000000;RESP_TM_VAR=49;RETRANS_SEGS=0;START_TM='2004-05-27-03.29.46. 000000';REMOTE_NODE_IP='9.155.177.145';REM_WINDOW_CNT=1;APP_JOB_NM='DFSKERN ';LAST_ACTIV msmt_id=c1c710dc090c0420000000fcda6eae44; timestamp=2004-05-31 05:09:48.0; adapter_host=wtsc61; hostname=wtsc61.itso.ibm.com;
ENTER A (ACTION) OR DM (DETAIL MENU) ??? CMD==> Figure 7-13 Threshold rearmed event
180
7.3.3 Automating NetView alert

This section provides some samples to perform automation on IBM Tivoli Monitoring for Network Performance events. As discussed in 7.2.3, Sample event automation program on page 173, there are only two event classes defined for IBM Tivoli Monitoring for Network Performance: threshold exceeded event and threshold rearmed event. For NPDA alert, this event content is further hidden in the sub-vector 31 for the original TEC event, as shown in Figure 7-12 on page 179. This provides a challenge on writing the automation program. We make a sample Message Automation Table (MAT) called ITMNP, as shown in Example 7-7. This MAT is then inserted to the active processing using the AUTOTBL MEMBER=ITMNP,FIRST command.
Example 7-7 ITMNP sample MAT * * ITMNP Event Automation starts here * IF MSUSEG(0000) = '' THEN BEGIN; IF HIER(3) = 'ITMNP APPL' THEN BEGIN; * Common processing for ITMNP events EXEC (CMD('ITMNPCMD COMMON') ROUTE(ONE +AUTO1)) CONTINUE(Y); * * ITMNP_ThresholdExceeded event * IF MSUSEG(0000.31(1).30) = . 'ITMNP_ThresholdExceeded' . THEN EXEC (CMD('ITMNPCMD DOWN') ROUTE(ONE +AUTO1)) CONTINUE(N); * * ITMNP_ThresholdRearmed event * IF MSUSEG(0000.31(1).30) = . 'ITMNP_ThresholdRearmed' . THEN EXEC (CMD('ITMNPCMD UP') ROUTE(ONE +AUTO1)) CONTINUE(N); END; END;
The automation called the program ITMNPCMD. This is a REXX program that will parse the actual event content and perform some processing. This sample ITMNPCMD program is provided in Example 7-8 on page 182.
181
Example 7-8 ITMNPCMD sample REXX program /*REXX*/ /*ITMNPCMD - Automation command to process ITMNP events */ /* Step 1 event_class parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value parse value Collect MSU information */ = MSUSEG('0000.31(1).30',2) MSUSEG('0000.31(2).30') with '=' monitor_host MSUSEG('0000.31(3).30') with '=' source MSUSEG('0000.31(4).30') with '=' message MSUSEG('0000.31(5).30') with '=' field_name MSUSEG('0000.31(6).30') with '=' msmt_detail MSUSEG('0000.31(7).30') with '=' msmt_id MSUSEG('0000.31(8).30') with '=' timestamp MSUSEG('0000.31(9).30') with '=' adapter_host MSUSEG('0000.31(10).30') with '=' hostname MSUSEG('0000.31(11).30') with '=' adapter_ip MSUSEG('0000.31(12).30') with '=' origin MSUSEG('0000.31(13).30') with '=' thold_id MSUSEG('0000.31(14).30') with '=' sub_source MSUSEG('0000.31(15).30') with '=' field_value MSUSEG('0000.31(16).30') with '=' severity
';' ';' ';' ';' ';' ';' ';' ';' ';' ';' ';' ';' ';' ';'
/* Step 2 - process argument */ parse arg cmd data select when cmd='COMMON' then do /* common processing */ say 'ITMNP event received for' message 'from' monitor_host end when cmd='UP' then do /* UP processing */ end when cmd='DOWN' then do /* DOWN processing */ end otherwise nop end exit
182
Chapter 8.
Historical reporting
This chapter discusses the interface of IBM Tivoli Monitoring for Network Performance that is used to store historical information. The interface collects the historical data that is then stored in Tivoli Data Warehouse. The discussion consists of the following topics: 8.1, Tivoli Data Warehouse overview on page 184 8.2, Tivoli Data Warehouse setup on page 184 8.3, Installation of the Warehouse Enablement Pack on page 187 8.4, ETL processing on page 197 8.5, Sample reports on page 202
183
8.1 Tivoli Data Warehouse overview

Tivoli Data Warehouse is a strategic Tivoli product that collects system management information and stores it in a central data warehouse. This data is then deployed to specific data marts for reporting purposes. For more information on Tivoli Data Warehouse V1.2, refer to Implementing Tivoli Data Warehouse V1.2, SG24-7100. Tivoli Data Warehouse uses data collected from each Tivoli product and loads it using a set of Extract, Transform, Load (ETL) programs. Each product provides the definition on how Tivoli Data Warehouse should handle their data; this is called the Warehouse Enablement Pack (WEP). The processing is shown in Figure 8-1.
Source data
ETL1
ETL2
CDW
Data mart
Figure 8-1 Warehouse processing
The WEP contains the following definitions: Database schema to be created in the Central Data Warehouse (CDW) Database schema to be created in the data mart ETL definitions that load data from the source to CDW ETL definitions that load data from CDW to the data mart Tivoli Data Warehouse V1.2 now supports a data source residing on DB2 for z/OS. The discussion in this chapter will see how IBM Tivoli Monitoring for Network Performance data is loaded to Tivoli Data Warehouse.
8.2 Tivoli Data Warehouse setup

Tivoli Data Warehouse consists of the following components: Tivoli Data Warehouse control server Tivoli Data Warehouse agent site Tivoli Data Warehouse reporting server, with Crystal Enterprise 9 We put the whole warehouse installation into a single machine, as we only want to evaluate the historical data collected from IBM Tivoli Monitoring for Network
184
Performance. We have two scenarios set up; therefore, we have two machines for Tivoli Data Warehouse: phoenix kingston This warehouse collects data from the AIX Web application server on jakarta. This warehouse collects data from the z/OS Web application server on SC61.
In Tivoli Data Warehouse V1.2, the z/OS data source requires that the CDW and data mart reside on z/OS. The configuration that we use is shown in Figure 8-2.
ETL1
ITMNPDB
IBM Tivoli Monitoring for Network Performance V1.2 WebSphere Application Server 5 DB2 V8.1 jakarta
TWH_CDW
E T L 2
TWH_MART
Tivoli Data Warehouse V1.2 Crystal Enterprise 9 DB2 UDB V7.2.10 Microsoft IIS server phoenix
ETL1
ITMNPDB
IBM Tivoli Monitoring for Network Performance V1.2 WebSphere Application Server 5 DB2 V8.1 SC61
TWH_CDW
E T L 2
TWH_MART
Tivoli Data Warehouse V1.2 Crystal Enterprise 9 DB2 UDB V7.2.10 Microsoft IIS server kingston
Figure 8-2 Warehouse configuration
The installation process for phoenix and kingston are different. The next sections discuss the installation on each. We only highlight the important steps on the Warehouse Enablement Pack.
Chapter 8. Historical reporting
185
8.2.1 Installation for distributed data source

The installation of the Tivoli Data Warehouse on phoenix is performed after we have the prerequisites installed. These are our existing platforms: Windows 2000 Server Service Pack 4 with Microsoft Internet Information Server (IIS) installed. DB2 Universal Database Version 7.2.10 with the data warehousing tools installed. The default database for DB2 warehouse control database is called DWCTRLDB; if you choose the typical install, this is the database that will be used. Tivoli Data Warehouse uses a control database called TWH_MD. We install Crystal Enterprise 9 first. The installation will customize the Microsoft IIS Web server and creates the necessary Crystal Enterprise 9 resources. Crystal Enterprise needs to be installed before you install Tivoli Data Warehouse. Note: If you use IBM HTTP Server, these customizations are not automatic. You need to use the IBM HTTP Administration Server to customize the configuration. We install Tivoli Data Warehouse using the quick start deployment. This installs all components of the Tivoli Data Warehouse V1.2 into a single machine. While this may not be the recommended configuration for a production environment, it is sufficient for our test environment. The databases TWH_CDW, TWH_MART, and TWH_MD are created and populated in the local machine phoenix. Now we are ready to install the warehouse enablement pack.
8.2.2 Installation for mainframe data source

To use data sources and databases on the z/OS platform, we need to install Tivoli Data Warehouse V1.2 using a customized install. Again, we need to have the prerequisites all installed. These are our existing platforms: Windows 2000 Server Service Pack 4 with Microsoft Internet Information Server (IIS) installed. DB2 Universal Database Version 7.2.10 with the data warehousing tools installed. The default database for DB2 warehouse control database is called DWCTRLDB; if you choose the typical install, this is the database that will be used. Tivoli Data Warehouse uses a control database called TWH_MD. We install Crystal Enterprise 9 first. The installation will customize the Microsoft IIS Web server and creates the necessary Crystal Enterprise 9 resources. Crystal Enterprise needs to be installed before you install Tivoli Data Warehouse.
186
We prepare the z/OS database using the following steps: 1. Create the necessary storage group and tablespaces for Tivoli Data Warehouse on DB2 for z/OS. We use our DB8D subsystem and invoke a SQL program using file input (SPUFI) from the DB2 interactive menu in ISPF. 2. The statement that we execute is shown in Example 8-1.
Example 8-1 Statement to initialize DB2 for z/OS CREATE STOGROUP TDWSG VOLUMES (TIVO01,TIVO02,TIVO03) VCAT DB8DU; CREATE DATABASE TWHTEMP BUFFERPOOL BP0 STOGROUP TDWSG AS TEMP; CREATE TABLESPACE TTEMP IN TWHTEMP USING STOGROUP TDWSG PRIQTY 5000 SECQTY 2000
The Tivoli Data Warehouse installation on kingston uses the Custom or Distributed installation types. In the installation wizard, we need to specify the z/OS databases as the Central Data Warehouse and data mart.
8.3 Installation of the Warehouse Enablement Pack

This section discusses the Warehouse Enablement Pack installation. The discussion is divided into the following topics: 8.3.1, Back up TWH databases on page 187 8.3.2, Warehouse Enablement Pack installation on page 188
8.3.1 Back up TWH databases

Before installing any warehouse enablement pack, we highly recommend backing up all the warehouse databases to ensure that you can return to a known valid state if you encounter an unrecoverable error during installation. We need to back up three databases: TWH_CDW, TWH_MART, and TWH_MD. See 4.6, Backup and recovery on page 91 for the DB2 distributed backup procedure and see 5.5, Backup and recovery on page 119 for the z/OS DB2 database backup procedure. Remember to stop the Microsoft Internet Information Server, Crystal Enterprise, Warehouse Server, and Warehouse Logger before you perform the backup process.
187
8.3.2 Warehouse Enablement Pack installation

To install the IBM Tivoli Monitoring for Network Performance warehouse pack, we completed the following steps on the control server: 1. Click Start Programs TDW Install a Warehouse Pack; this starts the warehouse pack installation wizard shown in Figure 8-3. In the Welcome window, click Next.
Figure 8-3 Install a Warehouse Pack window
2. The window in Figure 8-4 on page 189 shows the location of the Tivoli common logging directory, which will contain all the TDW log files. In our installation, we use the default location C:\Program Files\ibm\Tivoli\common. Click Next.
188
Figure 8-4 Tivoli Common Logging Directory window
3. In the window shown in Figure 8-5, click Add to add the IBM Tivoli Monitoring for Network Performance warehouse enablement pack.
Figure 8-5 Add Warehouse Pack window
4. In the location of the installation properties file window, shown in Figure 8-6 on page 190, click Browse to point to the location of the IBM Tivoli Monitoring
189
for Network Performance warehouse enablement pack installation properties file, twh_install_props.cfg. You can find this file in the IBM Tivoli Monitoring for Network Performance media. Click Next.
Figure 8-6 Location of installation properties
5. The installer will prompt for the data mart database to be used by the processes of the IBM Tivoli Monitoring for Network Performance warehouse enablement pack. It also prompts for the remote agent site that will run the ETL2 processes, as shown in Figure 8-7 on page 191.
190
Figure 8-7 Data mart and remote agent site settings
6. The installer will prompt for the central data warehouse database to be used by the processes of the IBM Tivoli Monitoring for Network Performance warehouse enablement pack. It also prompts for the remote agent site that will run the ETL1 processes At this time, you can opt to perform the scheduling settings for the ETL1 processes. If you opt to do so, the installation process will schedule the processes to run at the specified time and promote the processes to production status. You can also opt not to schedule the ETLs at installation time and perform the above tasks manually. The installer will also define the user authority for each one of the warehouse source and target processes. Click Next. See Figure 8-8 on page 192.
191
Figure 8-8 Central data warehouse and remote agent site settings
7. The installer will investigate the existence of an IBM Tivoli Monitoring for Network Performance ODBC connection. Click on Edit to specify the ODBC settings, as shown in Figure 8-9.
Figure 8-9 Editing IBM Tivoli Storage Manager ODBC settings
192
8. As shown in Figure 8-10, set the User ID of the IBM Tivoli Monitoring for Network Performance database Administrator and Password. Click New to create an ODBC to create a new connection.
Figure 8-10 IBM Tivoli Monitoring for Network Performance ODBC Settings
9. As shown in Figure 8-11 on page 194, create a connection name to the IBM Tivoli Monitoring for Network Performance sever host name; we used the name itmnp21, the default port of 50000, and the database name of itmndb. Click on OK to create the ODBC connection.
193
Figure 8-11 Installation menu window
10.As shown in Figure 8-12, the installer will test the ODBC connection and return to the ODBC Data Source Properties Panel. Click Next.
Figure 8-12 IBM Tivoli Monitoring for Network Performance ODBC Settings
194
11.The installation menu window shown in Figure 8-13 now lists the IBM Tivoli Monitoring for Network Performance warehouse enablement pack. Select this option and click Next to continue.
Figure 8-13 IBM Tivoli Monitoring for Network Performance install summary
12.Click Install in the summary window shown in Figure 8-14 on page 196 to start the IBM Tivoli Monitoring for Network Performance warehouse enablement pack installation.
195
Figure 8-14 Install panel
13.The final installation window contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard. See Figure 8-15.
Figure 8-15 Installation completed
196
Your Warehouse Enablement pack is ready.
8.4 ETL processing

The IBM Tivoli Monitoring for Network Performance extracts historical information from the IBM Tivoli Monitoring for Network Performance DB2 databases and loads it into the Tivoli Data Warehouse V1.2 environment. Subsets of the historical information stored in the central data warehouse database are loaded into data mart databases to enable reporting in specific areas of interest.
8.4.1 ETL overview

The IBM Tivoli Monitoring for Network Performance enablement pack includes report information related to the following: FTP performance statistics Open System Adapter Performance performance statistics and interface statistics TCP Application performance Reports TN3270 Performance Reports that use the SNMP measurements are not provided; however, the information can be displayed by the Web application. IBM Tivoli Monitoring for Network Performance generates reports for 22 categories of collected data. Sub reports of each category can be generated. There are over 100 sub reports that can be generated and approximately 10 configurable parameters can be specified for each sub report. You can also use the default values for all parameters except the measurement data start and stop dates. All reports can be generated for hourly, daily, weekly, monthly, quarterly, and yearly time frames. Reports are generated for all monitored resources. Resources are displayed as lines in a time-series graph. Alternatively, you can generate the top 10 sub reports that use a bar graph to compare the top 10 monitored resources. Table 8-1 provides a list of the warehouse objects installed with the IBM Tivoli Monitoring for Network Performance WEP.
Table 8-1 FNP WEP object names Warehouse Targets Warehouse Sources FNP_TWH_CDW_Target FNP_TWH_MART_Target FNP_TWH_CDW_Source FNP_TWH_MART_Source
197
Subject Area Processes Steps
FNP_IBM_Tivoli_MonitoringNetworkPerformance_V2.1.0.0 _Subject_Area FNP_c05_ETL1_Process FNP_m05_ETL2_Process FNP_c05_s010_extractzosdata FNP_c05_s020_transformzosdata FNP_c05_s030_loadzosdata FNP_c05_s040_extracticmpdata FNP_c05_s050_transformicmpdata FNP_c05_s060_loadicmpdata FNP_c05_s070_extractsnmpdata FNP_c05_s080_transformsnmpdata FNP_c05_s090_loadsnmpdata FNP_c05_s100_lastetlrun FNP_m05_s010_extract FNP_m05_s020_load FNP_m05_s*_<function>_rollup (step 30 - 280) FNP_m05_s300_prune_rollup
Steps
Table 8-1 on page 197 shows that the IBM Tivoli Monitoring for Network Performance warehouse enablement pack has the following processes: FNP_c05_ETL1_Process FNP_m05_ETL2_Process These processes are in the main central data warehouse ETL. It extracts information from an IBM Tivoli Monitoring for Network Performance server and loads it into the central data warehouse database. We scheduled the process to run once each 24 hour period to collect information at 02:00. You should choose a time of low activity on the IBM Tivoli Monitoring for Network Performance server in which to run this process. For example, you might choose to schedule the process in the early morning, after your nightly backups have completed but before the daily server maintenance processes begin. In order to obtain the steps and dependencies of the ETL processes, perform the following steps: 1. Start the IBM DB2 Control Center utility by selecting Start Programs IBM DB2 Control Center. 2. On the IBM DB2 Control Center utility, start the IBM DB2 Data Warehouse Center utility by selecting Tools Data Warehouse Center. The Data Warehouse Center logon window appears.
198
3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID, in our case, db2admin. 4. In the Data Warehouse Center Window, expand Subject Areas Processes and double click on FNP_c05_ETL1_Process. A similar process can be followed for the FNP_m05_ETL2_Process.
8.4.2 Testing, scheduling, and promoting the ETLs

Tivoli Data Warehouse V1.2 has simplified the installation process of warehouse enablement packs to the extent that the scheduling and promoting of ETLs developed for Version 1.2 are set during the warehouse enablement pack installation time. We verified the ETL processes manually by processing each step in sequence. The installation of the warehouse enablement pack installed all the processes in production mode. In data warehouse you can only run processes manually in test mode. This is how we verified that the ETL processes would run successfully; you can verify data collection on the data mart database as follows: 1. On the control server, start the DB2 Control Center tool by selecting Start Programs DB2 Control Center. On the DB2 Control Center, select Tools Data Warehouse, as shown in Figure 8-16.
Figure 8-16 Logon to Data Warehouse
2. The Data Warehouse logon screen will be presented. Click on the Advanced button to ensure that you are pointing to the correct database alias and warehouse server. Enter your DB2 administrator user ID and password. See Figure 8-17 on page 200.
199
Figure 8-17 Data Warehouse logon screen
3. The Data warehouse center will show the Subject Areas, Source Data warehouse databases, Target warehouse databases, Warehouse schemas, and the Administration options. Expand the Subject Areas FNP_IBM_Tivoli_MonitoringforNetworkPerformance_V2.1.0.0_Subject_ Area -> FNP_c05_ETL1_Process. 4. The steps for the FNP_c05_ETL1_Process will be listed in name order, which is also the order the steps will process when they run in production. The Mode column reflects that each step is in Production mode. We manually changed the steps to test mode by clicking on the first step and holding down the shift and the down tab arrow to select the FNP_c05* steps we were going to test. Right-click and select Mode Test, which will change all the steps to test mode, as shown in Figure 8-18.
Figure 8-18 Change FNP_c05_ETL1_Process steps to test mode
200
5. The mode for each step will change to Test. We right-click on the first process, FNP_c05_s010_extractzosdata, and click on Test, which will test the step. A process panel will appear and then disappear if the step executes successfully. 6. If the step executes unsuccessfully, you must investigate the error by going to the Warehouse Work in Progress menu. 7. Find the process that you run, right-click, and select Show log to diagnose the problem with each step. See Figure 8-19.
Figure 8-19 Show log of steps
8. Follow this process for each step to ensure the steps would run successfully when they were run in Production mode. 9. Once all the steps are verified to normal completion for the FNP_c05_ETL1_Process, the same process is used for the FNP_m05_ETL2_Process. 10.At this point, we change all the processes back to Production mode by manually changing the steps to production mode by clicking on the first step and holding down the Shift key and the down tab arrow key to select all 10 steps we are going to promote to production. Once the steps are highlighted, we expand Selected Mode Production, which will change all the steps back to Production mode.
201
8.5 Sample reports

The reports provided by IBM Tivoli Monitoring for Network Performance Warehouse Enablement Pack are installed directly in the Crystal Enterprise server. This section shows the additional configuration and use of these reports. The discussion is divided into the following topics: 8.5.1, Configuring Crystal Enterprise on page 202 8.5.2, Available reports on page 208 8.5.3, Accessing the Crystal ePortfolio on page 209
8.5.1 Configuring Crystal Enterprise

First, you need to have the data mart database TWH_MART be available as an ODBC data source at the Crystal Enterprise machine. In our environment, the Tivoli Data Warehouse and Crystal Enterprise are installed on the same machine, so that had already been set. In the IBM Tivoli Monitoring for Network Performance Version 2.1 README File (the known limitation section), there is a set of steps to set up the connection to the data mart from Crystal Enterprise. We include the steps here. If these steps are not completed, you will encounter errors while viewing the reports. The following instructions tell you how to set the Data Mart database connection in Crystal Management Console after installing and running the IBM Tivoli Monitoring for Network Performance Warehouse Enablement Pack. 1. Open the Crystal Management Console. Select Start Programs Crystal Enterprise 9 Crystal Launchpad, as shown in Figure 8-20 on page 203. Click on Crystal Management Console.
202
Figure 8-20 Crystal Enterprise Launchpad
2. Enter the Crystal administrator user ID and password to log onto the Crystal Management Console, as shown in Figure 8-21 on page 204. Use the Windows administrator user ID and its password. Click Log On.
203
Figure 8-21 Crystal Management logon
3. In the Crystal Management Console, click on the Manage Objects icon, as shown in Figure 8-22 on page 205.
204
Figure 8-22 Select Manage Objects
4. A list of 220 reports will be presented, which consists of 22 reports in 10 different languages. In our case, we used the English version of the reports, so we modified only the English reports. We clicked on the English CSMStorageReports first. The reports attributes are shown in Figure 8-23 on page 206; click the Database tab.
205
Figure 8-23 CSMStorageReport attributes
5. Distributed report only: Set the default information for logging onto the data source. We check the Automatically whenever the report is run option and entered the DB2 administrator user ID and password and then clicked Update. See Figure 8-24 on page 207.
206
Figure 8-24 Set the default information for logging onto the data source
6. Mainframe reports only: Set the default information for logging onto the data source. We checked Use Custom Database logon information specified here and entered the user ID and password for the Data Mart database on z/OS and then clicked Update. See Figure 8-24.
207
Figure 8-25 Set the default information for logging onto the data source
7. This process must be performed for each of the 22 reports that you will be viewing. Click on your browsers Back tab and follow this process for the rest of the reports. Once this is complete, you will be able to view the reports.
8.5.2 Available reports

Here is a list of predefined reports shipped with the IBM Tivoli Monitoring for Network Performance V1.2 warehouse enablement pack provides the following reports that can be used for workload analysis, capacity planning, and trend analysis: TCP Applications Workload TCP Connections Workload
208
UDP Applications Workload Response Time TN3270 Server TN3270 Client TN3270 Application OSA Adapter Port Status OSA Adapter Processor Utilization and Throughput OSA Ethernet Throughput Interface FTP Server FTP Server User FTP Client FTP Client User Enterprise Extender Availability Enterprise Extender Throughput and Traffic TCP Layer Stack IP Layer Stack UDP Layer Stack TCPIP Stack Memory CSM Storage Reports can be viewed in a variety of ways. For example, you can view bar or line graphs calculated on an hourly, daily, monthly, or yearly basis. Crystal Enterprise Professional Version 9 for Tivoli has, in comparison to a full Crystal license, reduced configuration options. If the reports shipped with IBM Tivoli Monitoring for Network Performance warehouse enablement pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.
8.5.3 Accessing the Crystal ePortfolio

All reports provided by Tivoli Data Warehouse V1.2 must be accessed using the ePortfolio feature of the Crystal Enterprise. The ePortfolio can be accessed from the Crystal Enterprise launchpad. 1. In order to access the Crystal Enterprise launchpad, we started an Internet browser and opened the following URL:
http://phoenix/crystal/enterprise9
The resulting Web page is shown in Figure 8-20 on page 203. 2. Select ePortfolio link and log in using the user ID of tivoli and a blank password, as shown in Figure 8-26 on page 210.
209
Figure 8-26 Crystal Enterprise - ePortfolio log on
3. After entering the required data, select Log On to proceed. Instead of No Folders in the guest users ePortfolio window in Figure 8-27 on page 211, there is now a folder for IBM Tivoli Monitoring for Network Performance reports.
210
Figure 8-27 Crystal Enterprise 9 - Folders
4. Go into the IBM Tivoli Monitoring for Network Performance folder and see the reports listed there, as shown in Figure 8-28 on page 212.
211
Figure 8-28 Crystal Enterprise 9 - available reports for ITSM
5. In order to obtain a report, select the desired report, for example, TCP_LayerStackReports, and select View, as shown in Figure 8-29 on page 213.
212
Figure 8-29 View TCP_LayerStackReports
6. The view report will launch and provide The report you requested requires further information panel. See Figure 8-30 on page 214.
213
Figure 8-30 Crystal Enterprise 9 - parameters
7. The time parameters must be entered in the exact format of DateTime(yyyy,mm,dd,hh,mm,ss). Enter the appropriate values and press Enter to see the TCP_LayerStackReports report. See Figure 8-31 on page 215.
214
Figure 8-31 Viewing reports
8. In the upper right corner of the screen, there will be a list of the z/OS operating systems that are collecting data. You can click on any of the operating systems and it will provide the details for the instances on the graph. See Figure 8-32 on page 216.
215
Figure 8-32 Details for TCP_LayerStackReports graph
216
Appendix A.
ITMNP configuration files and test programs

This appendix provides the configuration files we customized for our environment to set up IBM Tivoli Monitoring for Network Performance. The configuration files are provided for both AIX and z/OS Web application server. This appendix consists of four files: AIX itmnp.properties on page 218 z/OS itmnp.properties on page 218 Sample server TSO REXX program on page 219 Sample object REXX client program on page 220
217
AIX itmnp.properties
Example A-1 shows a sample itmnp.properties file for a monitor in the SC62 host connecting to AIX Web application server. We have stripped out the comments in the file.
Example: A-1 AIX itmnp.properties monitor_hostname=wtsc62.itso.ibm.com local_httpport=9455 WAS_hostname=jakarta.itsc.austin.ibm.com WAS_httpport=9454 DBName=ITMNPDB DBUserName=db2inst1 DBPassword=****** DBHostName=jakarta.itsc.austin.ibm.com DBPort=50000 DBCacheDirectory=/var/ibm/tivoli/common/FNP db2Security=CLEAR_TEXT_PASSWORD_SECURITY WebsphereServletProtocol=https trustStoreName=/etc/itmnp/itmnpMonitorTrustStore.jks trustStorePassword=WebAS keyStoreName=/etc/itmnp/itmnpMonitorKeyStore.jks keyStorePassword=WebAS keyManagerPassword=WebAS monitor.trace.level=DEBUG_MIN
z/OS itmnp.properties
Example A-2 shows a sample itmnp.properties file for a monitor in the SC61 host connecting to z/OS Web application server. We have stripped out the comments in the file.
Example: A-2 z/OS itmnp.properties monitor_hostname=wtsc61.itso.ibm.com WAS_hostname=wtsc61.itso.ibm.com DBName=DB8D DBUserName=TIVO01 DBPassword=TIVO01 DBHostName=wtsc61.itso.ibm.com DBPort=38030 DBCacheDirectory=/var/ibm/tivoli/common/FNP db2Security=CLEAR_TEXT_PASSWORD_SECURITY monitor.trace.level=DEBUG_MIN
218
Sample server TSO REXX program

Example A-3 shows our sample REXX server program that generates TCP/IP traffic to be monitored.
Example: A-3 Sample server TSO REXX program /*rexx*/ call Time(r) say Cab() Initializing sockets ... Time(e) say Cab() Socket Version Socket(VERSION) parse value Socket(INITIALIZE,RTMSRV,10,TCPIP) with rcinit message if rcinit <> 0 then call Error Error rcinit on sockinit function. say Cab() Socket initialized: message Time(e) parse value Socket(SOCKET,AF_INET,SOCK_STREAM,IPPROTO_TCP), with rcsocket socket if rcsocket < 0 then call Error Error on socksocket call rcsocket, socket say Cab() Socket socket initialized ... Time(e) x=Socket(SetSockOpt,socket,Sol_Socket,SO_REUSEADDR,on) address.family = AF_INET address.port = 7777 address.addr = INADDR_ANY rcbind = Socket(BIND,socket,address.family address.port address.addr) if rcbind <> 0 then call Error Error on bind rcbind say Cab() Bind on port address.port for socket socket ok rcbind, time(e) rclisten = Socket(LISTEN,socket,10) if rclisten <> 0 then call Error Error on listen rclisten say Cab() Listen on port address.port for socket socket ok, rclisten time(e) count = 1 t = 60 call Time(r) socketr = socket do forever say Cab() Selecting socket socket for t seconds ... count, time(e) parse value Socket(SELECT,READ socketr,t) with, rcselect nsock READ socket WRITE 1 message socket = Strip(socket) say Cab() Select rc rcselect nsock socket select when nsock = 0 then say Cab() Timeout ocurred ... Time(e) when nsock > 0 then call PROCESS otherwise call Error Error on select rcselect nsock end
Appendix A. ITMNP configuration files and test programs
219
end CLOSE: rcclose = Socket(CLOSE,socket) say Cab() Closed socket socket rcclose Time(e) exit 0 ERROR: parse arg message say Cab() message signal close CAB: return date() Time() PROCESS: parse value Socket(ACCEPT,socket) with rcaccept socket, addressaccept.domain addressaccept.port addressaccept.addr 1 message if rcaccept < 0 then call ERROR Error on accept message say Cab() Connected to addressaccept.addr addressaccept.port parse value Socket(RECV,socket,1024) with rcreceive len bufrecv if rcreceive <> 0 then call ERROR Error on receive rcreceive, len bufrecv say Cab() Received rcreceive rc len count=count+1 parse value Socket(SEND,socket,bufrecv) with message say Cab() Closing the socket socket rcclose = Socket(CLOSE,socket) return
Sample object REXX client program

Example A-4 shows our sample REXX client program that generates TCP/IP traffic to be monitored. Example: A-4 Sample object REXX client program
/* */ rc1 = RxFuncAdd(SockLoadFuncs,rxsock,SockLoadFuncs) rc2 = SockLoadFuncs() say Cab() Initializing sockets ... rc1 rc2 rcinit = SockInit() if rcinit <> 0 then call Error do forever
Error rcinit on sockinit function.
220
socket = SockSocket(AF_INET,SOCK_STREAM,IPPROTO_TCP) if socket < 0 then call Error Error on socksocket call socket say Cab() Socket socket initialized ... address.family = AF_INET address.port = address.addr = INADDR_ANY rcbind = SockBind(socket,address.) say Cab() Binding on port address.port for socket socket if rcbind <> 0 then call Error Error on bind rcbind say Cab() Binded on port address.port for socket socket Server_List = wtsc52 wtsc61 wtsc62 wtsc66 Server_Name = Word(Server_List,Random(1,Words(Server_list))).itso.ibm.com rc = SockGetHostByName(Server_Name,host.) address.family = AF_INET address.port = 7777 address.addr = host.addr say Cab() Connecting to port address.port for socket socket to address address.addr Server_Name rcconnect = SockConnect(socket,address.) if rcconnect <> 0 then say Cab() Error Error on connect rcconnect else do say Cab() Connected to port address.port for socket socket to address address.addr Server_Name dados = Substr(Copies(Date() Time(),50),100,Random(101,1000)) rcsend = SockSend(socket,dados) say Cab() Number of bytes sent on socket socket: rcsend end rcclose = SockSoClose(socket) say Cab() Closed socket socket rcclose time(e) timetowait=Format(1/Random(1,10),1,1) say Cab() Waiting for timetowait seconds Call SysSleep timetowait end Signal Drop ERROR: parse arg message errno = SockSock_Errno() say Cab() Error Number errno say Cab() message call SockPSock_Errno rcclose = SockSoClose(socket) DROP: call SysDropFuncs exit 0
Appendix A. ITMNP configuration files and test programs
221
CAB: return date() time()
222
Appendix B.
z/OS LDAP SDBM back end configuration files

This appendix provides the configuration files we customized for our environment to set up a z/OS LDAP server using the SDBM back end for WebShere Application Server on AIX. This appendix consists of three files: z/OS LDAP setup configuration file: ldap.profile on page 224 z/OS LDAP configuration file: SLAPDCNF on page 225 z/OS LDAP started procedure: LDAPSRV on page 226
223
z/OS LDAP setup configuration file: ldap.profile

This is the ldap.profile for our environment. This file includes the setup parameters for the SDBM and the TDBM back end. We only configured for the SDBM back end. We put a dummy value in the TDBM parameters to enable ldadcnf to run with return code 0. Example B-1 shows the content of this file. We removed the comments from the listing.
Example: B-1 Sample of z/OS LDAP setup file USR_LPP_ROOT='/' OUTPUT_DATASET='TIVO01.LDAP.OUTPUT1' OUTPUT_DATASET_VOLUME='TIVO01' TEMPORARY_DIR='/tmp' GLDHLQ='GLD' SGLDLNKVOL='SYSRES' GSKHLQ='GSK' SGSKLOADVOL='SYSRES' DSN_SDSNEXITHLQ='DB2' SDSNEXITVOL='TESTZZ' DSN_SDSNLOADHLQ='DB8D8' SDSNLOADVOL='TESTZZ' DSN_SDSNDBRMHLQ='DB8DU' DSN_SSID='DB8D' CEEHLQ='CEE' SCEERUNVOL='SYSRES' CBCHLQ='CBC' SCLBDLLVOL='SYSRES' CSSLIBVOL='SYSRES' LINKLIBVOL='SYSRES' LDAPUSRID='STC' LDAPUSRGRP='SYS1' TDBM_SUFFIX='o=ITSO' SDBM_SUFFIX='o=itso' ADMINDN='cn=LDAP Administrator' ADMINPW='secret' PROG_SUFFIX='62' APF_JOBCARD_1="//LDAPAPF APF_JOBCARD_2="// APF_JOBCARD_3="// APF_JOBCARD_4="/*JOBPARM APF_JOBCARD_5="" JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID," CLASS=A,MSGCLASS=T,MSGLEVEL=(1,1)," REGION=4096K,TIME=1440" SYSAFF=SC62,L=9999"
PRGCTRL_JOBCARD_1="//LDAPPRG PRGCTRL_JOBCARD_2="// PRGCTRL_JOBCARD_3="// PRGCTRL_JOBCARD_4="/*JOBPARM
JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID," CLASS=A,MSGCLASS=T,MSGLEVEL=(1,1)," REGION=4096K,TIME=1440" SYSAFF=SC62,L=9999"
224
PRGCTRL_JOBCARD_5="" DB2_JOBCARD_1="//LDAPDB2 DB2_JOBCARD_2="// DB2_JOBCARD_3="// DB2_JOBCARD_4="/*JOBPARM DB2_JOBCARD_5="" JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID," CLASS=A,MSGCLASS=T,MSGLEVEL=(1,1)," REGION=4096K,TIME=1440" SYSAFF=SC62,L=9999"
RACF_JOBCARD_1="//LDAPRAC RACF_JOBCARD_2="// RACF_JOBCARD_3="// RACF_JOBCARD_4="/*JOBPARM RACF_JOBCARD_5=""
JOB 'ACCOUNTING INFO',NOTIFY=&SYSUID," CLASS=A,MSGCLASS=T,MSGLEVEL=(1,1)," REGION=4096K,TIME=1440" SYSAFF=SC62,L=9999"
# The following include the advanced profile files. # Update the following lines to properly import advanced # configuration settings ${SOURCE_CMD} ${USR_LPP_ROOT}/usr/lpp/ldap/etc/ldap.slapd.profile ${SOURCE_CMD} ${USR_LPP_ROOT}/usr/lpp/ldap/etc/ldap.db2.profile ${SOURCE_CMD} ${USR_LPP_ROOT}/usr/lpp/ldap/etc/ldap.racf.profile
z/OS LDAP configuration file: SLAPDCNF

This is the HLQ.LDAP.OUTPUT(SLAPDCNF) file for our environment. Example B-2 shows the example of our configuration. This file includes the configuration for the SDBM back end, but it does not include the SSL security configuration.
Example: B-2 Sample of the SLAPDCNF file listen ldap://:3389 maxConnections 60 adminDN "cn=LDAP Administrator" adminPW "secret" logfile //'TIVO01.LDAP.OUTPUT(LOG)' # # SDBM-specific CONFIGURATION SETTINGS # database sdbm GLDBSDBM suffix "o=itso" sizeLimit 2000 timeLimit 180 # # TDBM-specific CONFIGURATION SETTINGS # Extended Operations (EXOP) Backend CONFIGURATION SETTINGS
Appendix B. z/OS LDAP SDBM back end configuration files
225
z/OS LDAP started procedure: LDAPSRV

This is a started procedure: LDAPSRV for our z/OS LDAP server. The DB2 load library has been removed from the STEPLIB of the procedure, as our environment does not use the TDBM back end, as shown in Example B-3.
Example: B-3 Sample of the LDAPSRV started procedure //LDAPSRV PROC REGSIZE=0M, //*-------------------------------------------------------------------//* CUSTOMIZABLE SYMBOLIC PARAMETERS //*-------------------------------------------------------------------// PARMS='', // PGLDHLQ='GLD', // PCNFOUT='TIVO01.LDAP.OUTPUT', // OUTCLASS='A' //*-------------------------------------------------------------------//GO EXEC PGM=GLDSLAPD,REGION=&REGSIZE,TIME=1440, // PARM=('/&PARMS >DD:SLAPDOUT 2>&1') //*-------------------------------------------------------------------//STEPLIB DD DSN=&PGLDHLQ..SGLDLNK,DISP=SHR //* //*-------------------------------------------------------------------//* CONFIG can be used to specify the LDAP server config file. //*-------------------------------------------------------------------//CONFIG DD DSN=&PCNFOUT.(SLAPDCNF),DISP=SHR //*-------------------------------------------------------------------//* ENVVAR can be used to specify any environment variables //* If this DD is used, the name of the data set must be customized //* for the installation. //*-------------------------------------------------------------------//ENVVAR DD DSN=&PCNFOUT.(SLAPDENV),DISP=SHR //SLAPDOUT DD SYSOUT=&OUTCLASS //SYSOUT DD SYSOUT=&OUTCLASS //SYSUDUMP DD SYSOUT=&OUTCLASS //CEEDUMP DD SYSOUT=&OUTCLASS
226
Appendix C.
Tips collection
This appendix provides overview information, a collection of useful commands, tools, and utilities for products related to IBM Tivoli Monitoring for Network Performance. As discussed in Chapter 2, Components and architecture on page 7, IBM Tivoli Monitoring for Network Performance utilizes various IBM software products and interfaces to other software. This appendix can serve as a cheat sheet for implementers and administrators that want to work with IBM Tivoli Monitoring for Network Performance without needing to be an expert on the various products. The products discussed in this appendix are: WebSphere Application Server Version 5 on page 228 DB2 Universal Database Version 8 on page 229 z/OS Communication Server TCP/IP on page 233 IBM Tivoli NetView for z/OS Version 5 on page 233 IBM Tivoli Enterprise Console Version 3.9 on page 235 IBM Tivoli Data Warehouse Version 1.2 on page 237
227
WebSphere Application Server Version 5

WebSphere Application Server is IBMs premier Web application server. WebSphere Application Server Version 5 provides a single Java Virtual Machine (JVM) environment for all the Web applications. It conforms to the Java 2 Enterprise Edition (J2EE) specification. This appendix provides some quick tips and facts about WebSphere Application Server usage and operation. Some of this information applies to both z/OS and AIX based WebSphere Application Server; some information is different on those platforms.
Administration
WebSphere Application Server Version 5 is administered using a Web based application. This application conforms to the J2EE specification and manages the setting of the WebSphere Application Server. The application manages the enterprise applications under the WebSphere directory structure and also a set of configuration XML files. The administration is a J2EE application and runs inside the WebSphere Application Servers Java Virtual Machine. Sometimes, a configuration error causes WebSphere to abend in startup. This results in an inaccessible WebSphere Application Server. You have to either restore the WebSphere configuration file system or manually edit the configuration file to fix the error. This requires you to understand some important configuration files. The configuration files are stored in XML format. These files reside under the AppServer/config directory of the WebSphere root directory. WebSphere is organized in a cell - node - server structure. A base WebSphere configuration cell only consists of a single node, while a node deployment structure allows multiple nodes in a cell. A node is typically mapped to a single server machine. This structure allows multiple servers to reside in a single machine. The configuration scope conforms to this structure; each setting of an XML file applies to a configuration level. The following are some important configuration files that may need to be edited to recover from a failure in a WebSphere start up: security.xml (cell level) server.xml (server level) variables.xml (all levels)
228
Attention: In z/OS WebSphere server, these XML files are stored in ASCII, although the system is in EBCDIC. You cannot edit these files from the UNIX System Services. You need to transfer the files to an ASCII based system to edit them and transfer them back.
J2EE based application

There are some file types that are related to the J2EE based application; they are: Java archive file (jar) Web archive file (war) Enterprise application archive file (ear) Resource archive file (rar) An installed application resides under the AppServer/installedApps directory.
Startup and shutdown procedure

The startup and shutdown of WebSphere on a distributed platform is performed by using scripts from WebSphere/AppServer/bin. The scripts are: To start the server, use startServer.sh. To stop a server with security active, use stopServer.sh -username <uid> -password <pwd>. On z/OS, WebSphere is governed by a set of started tasks that runs UNIX System Services processes. These processes are typically grouped into the following: Controller process: This is the start up process for WebSphere for z/OS. Daemon process is the primary process that starts up the application server. Servant process is the Java Virtual Machine engine that serves the clients.
DB2 Universal Database Version 8

DB2 is IBMs premier relational database system. It resides on both the z/OS and distributed systems. This appendix discusses some quick tips on using DB2, either on z/OS or distributed.
Appendix C. Tips collection
229
Relational database system

DB2, as a relational database management system (RDBMS), organizes its information in tables, with their indexes. These tables are physically stored in files in the file systems. These file containers, where the table resides, are called tablespaces. On a z/OS platform, tablespaces are implemented as Linear VSAM files, while on distributed tablespaces that are associated with a path or file system. For performance reasons, tablespaces are logically organized in pages. Each page has a fixed size that can contain one or more rows of the table. A row must reside in a single page. Typically, the default page size is 4096 bytes or 4 KB. However, some tables has rows that are larger than this size, so there is a need to get larger pages, such as 16 KB or 32 KB. The access to the pages is cached using memory areas called the bufferpools. The bufferpools store pages and reuse them as request come in, in order to minimize actual disk I/O. Bufferpools have segments at the same size as the pages, so if the pages are 16 KB, the bufferpool must use 16 KB segments too. Programs access the database tables must be known to DB2. The programs access profile is stored in plans and packages. Plans for DB2 for z/OS contain a single executable or a collection of packages. Packages specify a module access profile to DB2. Most of the processing in DB2 programs are based on packages. The access profile is important for understanding the necessary access required for the program.
DB2 on z/OS systems

DB2 on z/OS acts as a subsystem. It consists of several started tasks: MSTR: The master database process DBM1: The database access process DIST: The distributed access thread manager SPAS: The stored procedure address space The following commands are some console commands that you can use. These commands need to be prefixed with the command identifier set for the DB2 system; we represent this prefix with a hyphen (-). You need to substitute this with the real prefix when issuing these commands. START DB2: This starts the DB2 system. The syntax of the command is -START DB2. STOP DB2: This stops the DB2 system. The syntax of the command is -STOP DB2 MODE(QUIESCE).
230
DISPLAY THREAD: This displays all the connected applications to the DB2 subsystem. The syntax of the command is -DIS THD(*). DISPLAY BUFFERPOOL: This displays the size and usage statistics of the buffer pools in DB2. The primary usage of this command is -DIS BPOOL(<bpname>). ALTER BUFFERPOOL: This command activates or resizes a DB2 bufferpool size. The primary usage of this command is -ALT BPOOL(<bpname>) VPSIZE(size).
DB2 on AIX or Window systems

On AIX systems, DB2 resides in instances. Each instance is owned by the DB2 instance owner. The instance is the individual DB2 process that can be started or stopped. The following are some useful commands that you can use: Starting the DB2 environment: In Windows, issue the DB2CMD command to establish the command window for DB2. In AIX, use the environment setting in $DB2idhome/sqllib/.db2profile. Running the DB2 command. Use the db2 command. SQL and DB2 commands can be executed using this command. On a UNIX platform, you may need to enclose the SQL statement between double quotes. Running the graphical administration tool using the db2cc command. This starts the DB2 Control Center Java application that lets you manage and configure DB2 databases. Managing the ODBC database connection using the db2cca command. This allows creation of an ODBC setting for DB2 databases.
Differences between z/OS and distributed DB2 processing

There are some differences between the processing of the DB2 database for z/OS and distributed. We discuss these differences in this section.
Instances
In DB2, an instance contains one or more databases. Each database in turn consists of tables and indexes. On z/OS, an instance is accessed in whole; no duplicates on tables or indexes are allowed. On distributed systems, a database is the unit of access. A remote connection in z/OS connects to an instance, while in distributed it connects to a database.
231
DB2 subsystem (z/OS)

database
DB2 instance (AIX/Windows)

database database tablespace table tablespace table tablespace table
tablespace table
tablespace table tablespace table
database tablespace table tablespace table
Distributed connection
Figure C-1 DB2 concepts
Tablespaces
A tablespace is a container that stores one or more tables. In z/OS, a tablespace is made from a single linear VSAM dataset. The organization of tables in the dataset is managed by DB2. In distributed, a tablespace is a container, path, or a file system. Distributed tables are files in that tablespace path. In z/OS, indexes are created in indexspaces, which are separate linear VSAM datasets; in distributed, indexes are files in the tablespace path.
Bufferpools
Bufferpools are used to buffer information from disk to boost performance. In z/OS, a set of bufferpools are already allocated; when we need to use any bufferpools, we change the size of an existing one (which is typically set to zero). In distributed, we can create new bufferpools with a specific page size and size.
Important tables
When working with a DB2 database, there are several tables that contain system information. These tables are called catalog tables. They are important in understanding the structure of the database. Some of these tables are: SYSIBM.SYSTABLES. Use the SQL statement:
SELECT CREATOR, NAME FROM SYSIBM.SYSTABLES
232
SYSIBM.SYSTABLESPACES. Use the SQL statement:

SELECT DBNAME, NAME FROM SYSIBM.SYSTABLESPACES
SYSIBM.SYSDATABASES. Use the SQL statement:

SELECT DBNAME FROM SYSIBM.SYSDATABASES
SYSIBM.SYSSTOGROUP and SYSIBM.SYSVOLUMES. Use the SQL statement:

SELECT A.NAME, B.VOLUMES FROM SYSIBM.SYSSTOGROUP A, SYSIBM.SYSVOLUMES WHERE A.NAME = B.NAME
z/OS Communication Server TCP/IP

This section provides some commands that you can issue from the z/OS operator console for modifying TCP/IP processing: Showing TCP/IP connection and port usage: D TCPIP,,NETSTAT,CONN. This command allows you to see which connections are made. Running an obey file: V TCPIP,,OBEYFILE,datasetname. This command allows a temporary override of the TCP/IP setting. Additional commands in a booklet format can be downloaded from:
http://www.redbooks.ibm.com/tips/TIPS0091/tips0091.pdf
IBM Tivoli NetView for z/OS Version 5

IBM Tivoli NetView for z/OS is the automation engine for z/OS systems. It support both systems and network automation. IBM Tivoli NetView for z/OS has the following features: Hardware monitor (NPDA) Session monitor (NLDM) Status monitor (STATMON) Program to program interface (PPI) Automation support Command interface Event automation system Graphical interface
233
Alerts structure
Alerts are historically used as an unsolicited information trigger when a network hardware detects a potential problem. The subsystem that works with alerts in IBM Tivoli NetView for z/OS is the hardware monitor, which is also called the network problem determination assistant (NPDA). Alerts are structured as a Network Management Vector Table (NMVT) (see SNA Formats, GA27-3136 for more information about NMVT). It consists of a primary vector, sub-vectors, and sub-sub vectors. The following are several of the well-known sub-vectors: Sub-vector 5: This sub-vector contains information about the hierarchy of the device that triggers this alert. Each hierarchy element includes the name and type of device that it represents. This sub-vector is typically used to identify the source of the alert. Sub-vector 00: Text message. Sub-vector 92: Generic alert data. Sub-vector 93: Probable cause. Sub-vector 33: User data. Sub-vector 31: Self-defining text messages, which are used for representing TEC slots values.
Automation support
Automation in IBM Tivoli NetView for z/OS is implemented using automation tables. The automation system is invoked whenever a message or alert is received by NetView, so the automation table is often referred to as the message automation table (MAT). The automation table consists of a set of IF - THEN rules that govern how message or alerts must be handled. Typically, a statement has the following format:
IF condition THEN action
The action can be a single command or a group of commands or statements between a BEGIN END block. Automation tables are managed by the AUTOTBL command. The AUTOTBL command allows testing, activation, insertion, and removal of automation table members.
234
Event automation service

The event automation service was introduced to provide an interface between IBM Tivoli NetView for z/OS and IBM Tivoli Enterprise Console. It provides three services: Converting hardware monitor alerts to TEC events Converting console messages to TEC events Converting TEC events to hardware monitor alerts The event automation service is one of the earliest NetView programs that require UNIX System Services. It converts TEC slots into alert sub-vectors and vice-versa. This service runs in its own address space, typically called IHSAEVNT. The event automation service communicates with IBM Tivoli NetView for z/OS using PPI channels. When the task starts, we can see that new PPI channels are established with the DISPPI command.
IBM Tivoli Enterprise Console Version 3.9

The IBM Tivoli Enterprise Console is an event processing application that is based on Tivoli Management Framework. Conceptually, the operation of any Framework based application can use the Tivoli Desktop or command line (CLI). The CLI operation must have the environment set first. The environment is set using the /etc/Tivoli/setup_env.sh (UNIX platform) or %WINDIR%\System32\drivers\etc\Tivoli\setup_env.cmd (Windows). This command needs to be invoked before any Tivoli CLI command can be run.
IBM Tivoli Enterprise Console structure

The conceptual structure of IBM Tivoli Enterprise Console is shown in Figure 8-33 on page 236.
235
Rulebase Event Server
TEC
Reception
Event Adapter Event source
Figure 8-33 Structure of IBM Tivoli Enterprise Console
In Figure 8-33, the main part is the event server. It receives events from various event sources. It uses a database using the Tivoli Frameworks RDBMS InterfaceModule (RIM) to access the TEC database. It processes events using a rulebase. Event sources generate events that are formatted by the event adapters, which send them to the tec_reception process. The rulebase consists of two main parts: the event definition and the event processing parts. The event definition consists of a set of baroc files. These files define events classes and the fields that they have. The fields are called slots. These events definitions are object oriented, which means that event definitions can be sub-classes, and the sub-class inherits the super-class attributes. The event processing part is defined as a set of ruleset files. These files are written in the prolog programming language. Each ruleset defines a set of rules that may be applied to incoming events and acts on them accordingly.
Important Tivoli Management Framework commands

The following are some commonly used commands: wlookup The wlookup command is used to get a list of all Tivoli object types using wlookup -R or to get all the objects of a specific type using wlookup -ar <objtype>.
236
wbkupdb
Tivoli database backup is performed using the command wbkupdb -d <path>, while restore is performed using the command wbkup -r -d <path>. This is a general command to interact with the Tivoli object dispatcher. For example, odadmin odlist retrieves all managed nodes and their statuses, while odadmin itself retrieves the current Tivoli server process information. This command retrieves all running Tivoli object methods and lists the last 200 object methods. The odstat command is very useful for debugging errors.
odadmin
odstat
Important IBM Tivoli Enterprise Console commands

The following are some CLI that we use with IBM Tivoli Enterprise Console. First, IBM Tivoli Enterprise Console is based on a rulebase. It governs how it reacts and processes events. wlsrb This command lists any defined rulebases in the system. The command wlsrb -d will show where the rulebases are stored. This command shows the currently active rulebase. The wrb is a shell script that invoke various rulebase processing commands. Some of the processing options are: This imports a new baroc file into the rulebase. This imports a new ruleset file into the rulebase. This compiles the rulebase into an internal form. This loads the rulebase as the current rulebase.
wlscurrb wrb
wrb -imprbclass wrb -imprbrules wrb -comprules wrb -loadrb
Other commands that are useful are: wstartesvr wstopesvr wtdumprl wtdumptr The wstartesvr command starts the event server. The wstopesvr command stops the event server. The wtdumprl command dumps the reception log. The wtdumptr command shows the automation processing log.
IBM Tivoli Data Warehouse Version 1.2

IBM Tivoli Data Warehouse is a historical data collection system that collects system management information from various IBM Tivoli applications. IBM Tivoli
237
Data Warehouse provides a consolidated platform for data collection in IBM Tivoli products. From this platform, it is very convenient to perform historical analysis, trend calculation, and other bulk processing.
Data warehouse concepts

IBM Tivoli Data Warehouse is based on the DB2 Version 7 warehouse system. In Version 1.2, the reporting portion is based on Crystal Enterprise 9. The conceptual structure of the IBM Tivoli Data Warehouse is shown in Figure 8-34.
TWH_MD Warehouse server DB2 server Reporting server (Crystal Enterprise)
source
TWH_CDW
TWH_MART
Figure 8-34 IBM Tivoli Data Warehouse structure
DB2 Version 7 warehouse runs on the Windows platform only (DB2 Version 8 supports AIX and other UNIX platforms, but IBM Tivoli Data Warehouse only supports DB2 Version 7). The warehouse server runs as two Windows services; Warehouse server (vwkernel) and Warehouse logger (vwlogger). In the warehouse processing, there are several databases that are involved. The names shown in Figure 8-34 are the typical names for the databases. TWH_MD This is the control database. It contains metadata information on how to process data from the source to the target database. The processing itself is often referred as the extract, transform, load (ETL) process. This is the central data warehouse. It stores all data from the source as indicated by the appropriate ETL program. The data in the warehouse is raw.
TWH_CDW
238
TWH_MART
This is the reporting database. It is written by the warehouse server from data extracted from the data warehouse. The data is extracted based on a certain reporting requirement. The reporting server accesses this database. Typically the tables are arranged in a star schema.
Warehouse enablement pack

Each IBM Tivoli application has data and information stored in a database. The warehouse enablement pack (WEP) contains the support information to integrate the application into IBM Tivoli Data Warehouse. Typically, the WEP consists of the following: Additional tables needed in the central data warehouse Additional tables needed in the data mart ETL programs to transfer data into the central data warehouse ETL programs to extract data from the central data warehouse into a reporting data mart Report definition for Crystal Enterprise The installation program will put each component into the IBM Tivoli Data Warehouse server.
239
240
Appendix D.
Underlying technology
This appendix discusses the basics of some underlying technologies and their application to IBM Tivoli Monitoring for Network Performance. These standards based technologies are considered to be nontraditional in z/OS. The discussion consists of the following topics: Light-weight Directory Access Protocol (LDAP) on page 242 eXtensible Markup Language (XML) on page 247 Certificates and encryption on page 251 Secure Sockets Layer protocol on page 261
241
Light-weight Directory Access Protocol (LDAP)

Information describing the various users, applications, files, printers, and other resources accessible from a network is often collected into a special database that is sometimes called a directory. As the number of different networks and applications has grown, the number of specialized directories of information has also grown. This growth results in islands of information that are difficult to share and manage. If all of this information could be maintained and accessed in a consistent and controlled manner, it would provide a focal point for integrating a distributed environment into a consistent and seamless system. LDAP is an open industry standard that has evolved to meet these needs. LDAP defines a standard method for accessing and updating information in a directory. LDAP is gaining wide acceptance as the directory access method of the Internet and is therefore also becoming strategic within corporate intranets. It is being supported by a growing number of software vendors and is being increasingly incorporated into applications. LDAP defines a message protocol used by directory clients and directory servers. The LDAP Protocol uses a variety of messages. For example, a bindRequest may be sent from the client to the LDAP server at the beginning of a connection. A searchRequest is used to search for a specific entry in the directory. There are also associated LDAP APIs for the C language and ways to access LDAP from within a Java application. Additionally, within the Microsoft development environment, you can access LDAP directories through its Active Directory Service Interface (ADSI). In general with LDAP, the client is not dependent upon a particular implementation of the server; the server can implement the directory however it chooses. LDAP defines a communication protocol. That is, it defines the transport and format of messages used by a client to access data in an X.500-like directory. LDAP does not define the directory service itself. When people talk about the LDAP directory, they refer to the information that is stored and can be retrieved by the LDAP protocol. The Comite Consultatif International Telephonique et Telegraphique or Consultative Committee on International Telephony and Telegraphy (CCITT), which is now called the International Telecommunications Union Telecommunication Standardization Sector (ITU-T), defined the X.500 standard in 1988. The X.500 standard then became International Standards Organization (ISO) 9594, Data Communications Network Directory, Recommendations X.500/X.521 in 1990, though it is still commonly referred to as X.500.
242
X.500 organizes directory entries in a hierarchical name space capable of supporting large amounts of information. It also defines powerful search capabilities to make information retrieval easier. Because of its functionality and scalability, X.500 is often used together with add-on modules for interoperation between incompatible directory services. X.500 specifies that communication between the directory client and the directory server use the Directory Access Protocol (DAP). However, as an application layer protocol, the DAP requires the entire Open Systems Interconnection (OSI) protocol stack to operate. Supporting the OSI stack requires more resources than are available in many small environments. Therefore, an interface to an X.500 directory server using a less resource-intensive or lightweight protocol (in this case, LDAP) is desired. An application client program initiates an LDAP message by calling an LDAP API. But an X.500 directory server does not understand LDAP messages. In fact, the LDAP client and X.500 server even use different communication protocols (TCP/IP versus OSI). The LDAP client actually communicates with a gateway process (also called a proxy or front end) that forwards requests to the X.500 directory server. This gateway is known as an LDAP server. It services requests from the LDAP client. It does this by becoming a client of the X.500 server. At the beginning, the LDAP server implementations supported both OSI and TCP/IP to be able to translate requests received by LDAP clients to DAP requests required to access X.500 directories. Newer LDAP server implementations, such as the IBM SecureWay Directory server, support only the LDAP protocol to access the directory. The LDAP server on the iSeries server is called Directory Services and implements the IBM SecureWay Directory. All modern LDAP directory servers are based on LDAP Version 3. You can use a Version 2 client with a Version 3 server. However, you cannot use a Version 3 client with a Version 2 server unless you bind as a Version 2 client and use only Version 2 APIs. All LDAP servers share many basic characteristics, since they are based on industry-standard Request for Comments (RFCs). However, due to implementation differences, they are not all completely compatible with each other when there is not a standard defined. If application developers could be assured of the existence of a directory service, then application-specific directories would not be necessary. However, a common directory must address the problems mentioned above. It must be based on an open standard that is supported by many vendors on many platforms. It must be accessible through a standard API. It must be extensible so that it can hold the types of data needed by arbitrary applications. Also, it must provide full functionality without requiring excessive resources on smaller
Appendix D. Underlying technology
243
systems. Since more users and applications will access and depend on the common directory, it must also be robust, secure, and scalable. When such a directory infrastructure is in place, application developers can devote their time to developing applications instead of application-specific directories. In the same way that developers rely on the communications infrastructure of TCP/IP and remote procedure call (RPC) to free them from low-level communication issues, they must be able to rely on powerful, full-function directory services. LDAP is the protocol to be used to access this common directory infrastructure. Like HTTP (hypertext transfer protocol) and FTP (file transfer protocol), LDAP has become an indispensable part of the Internet's protocol suite. When applications access a standard common directory that is designed in a proper way instead of using application-specific directories, redundant and costly administration can be eliminated and security risks are more controllable. For example, the telephone directory, e-mail, and Web applications shown below can all access the same directory to retrieve an e-mail address or other information stored in a single directory entry. The advantage is that the data is kept and maintained in one place. Various applications can use individual attributes of an entry for different purposes (assuming that the they have the correct authority). New uses for directory information will be realized, and a synergy will develop as more applications take advantage of the common directory. Figure D-1 on page 245 depicts the advantages of this arrangement.
244
Telephone Directory Application E-mail Application WebSphere Application Server A
WebSphere Application Server B
Directory
Objects 0=IBM CN=John CN=Wendy CN=Wolfgang sn (surname): Eckert telephoneNumber=2022 givenName (Firstname): Wolfgang uid (UserID): weckert userPassword: ******** mail (e-mail): wolf@iseries.ibm.com CN=Tom
HTTP Web Server
Figure D-1 Several applications using attributes of the same entry
Storing data in a directory and sharing it among applications saves you time and money by keeping administration effort and system resources down. Many IBM applications also utilize directories to centrally store and share information. The number of applications that support LDAP directories is constantly increasing. For example, LDAP directory support, such as for authentication and configuration management, is provided in various IBM Operating Systems, IBM WebSphere Application Server, WebSphere Portal Server, Tivoli Access Manager, Directory Server (Directory Server), IBM HTTP server, Lotus Domino, and so on. A directory contains a collection of objects organized in a tree structure. The LDAP naming model defines how entries are identified and organized. Entries are organized in a tree-like structure called the Directory Information Tree (DIT). Entries are arranged within the DIT based on their distinguished name (DN). A DN is a unique name that unambiguously identifies a single entry. DNs are made up of a sequence of relative distinguished names (RDNs). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the DIT to the directory entry. A DN is composed of a sequence of RDNs separated by commas, such as cn=thomas,ou=itso,o=ibm.
245
You can identify common names (CNs) within DNs. You also can organize entries, for example, after organizations. You can further split the tree into organizational units within a single organization as needed. You can define your DIT based on your organizational needs, as in the simple example below (Figure D-2). If you have, for example, one company with different divisions, you may want to start with your company name under the root as the organization (o) and then branch into organizational units (ou) for the individual divisions. In case you store data for multiple organizations within a country, you may want to start with a country (c) and then branch into organizations. Figure D-2 provides an example of this approach.
Directory Root (top)
o=IBM
c=US
ou=Marketing
ou=Support
o=ACMESupply
o=iSeriesShop
cn=mbarlen objectClass=Person objectClass=ePerson mail=marion@ibm.com sn=Barlen givenName=Marion telephoneNumber=112 cn=Klaus objectClass=Person objectClass=ePerson mail=Ktebbe@ibm.com sn=Tebbe
cn=tbarlen objectClass=Person objectClass=ePerson mail=thomas@acme.com sn=Barlen deviceID=PrinterSales objectClass=cimPrinter objectClass=ePrinter location=Printer room 3rd floor owner=John Doe queuename=lsprt01 maxCopies=10
Figure D-2 Example of a directory information tree
Each object also referred to as an entry in a directory belongs to one or more object classes. An object class describes the content and purpose of the object. It also contains a list of attributes, such as a telephone number or surname, that can be defined in an object of that class. You can publish entries of different object classes under another object. The object class also defines which of the attributes must be defined (are required) when creating an object of this class and which attributes are optional.
246
Object classes also can inherit characteristics, such as attributes from other object classes. In the example of the ePrinter, the class inherits all of the attributes that are defined in the class cimPrinter. Therefore, you must define the deviceID when you create an ePrinter object. Optionally, you can specify the location, owner, and queuePtr attribute of ePerson and all of the attributes of cimPrinter. Attributes themselves also have certain characteristics. The surname attribute name, for example, is defined as sn and surName and describes a person's family name. The attribute definition also specifies the syntax rules for the attribute value. A telephone number may only contain numbers and hyphens, while the surname consists of alpha characters. Other specifications include whether this attribute can contain only one or many values, the matching rules, the Object Identifier (OID), and so on. The IBM Tivoli Directory Server (ITDS) product also includes some IBM proprietary extensions for each attribute. Other manufacturers, such as Microsoft, have similar extensions. The IBM extensions also include an access class, which is used in combination with access control lists (ACLs) to control who can perform a certain action on the attribute value (such as read, write, search, or compare operations). All the objects and attributes with their characteristics are defined in schemas. The schema specifies what can be stored in the directory. Schema-checking ensures that all required attributes for an entry are present before an entry is stored. Schema-checking also ensures that attributes not in the schema are not stored in the entry. Optional attributes can be filled in at any time. A schema also defines the following: Inheritance Subclassing of objects Where in the DIT structure (hierarchy) objects may appear
eXtensible Markup Language (XML)

XML is a proposed standard by the World Wide Web Communities (W3C). It contains an extensible data structure for communicating structured information through a standard protocol, such as HTTP. The format of an XML document highly resembles a HTML or SGML document; however, the tags that are used are usually user defined. These user defined tag are defined in a Document Type Definition or DTD file. An XML document contains fields that is delimited with begin and end tags of the fields. A field can also contain a set of other fields. The XML is usually read by an XML parser. There are some commonly available XML parsers, such as SAX (Simple API for XML).
247
Figure D-3 shows an example of an XML document.
<?xml version="1.0" standalone="no"?> <HTTP-TRANSACTION> <REQUEST> <REQUEST-LINE NAME="STATIC"> <METHOD>GET</METHOD> <URI>/</URI> <PROT-VER>HTTP/1.0</PROT-VER> </REQUEST-LINE> <HEADERS> <HL>Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*</HL> <HL>Accept-Charset: iso-8859-1,*,utf-8</HL> <HL>Accept-Language: en</HL> <COOKIE> <INVOCATION> <FUNCTION NAME="retrieve_cookie"></FUNCTION> </INVOCATION> </COOKIE> <HL>Host: tokyo.itsc.austin.ibm.com</HL> <HL>User-Agent: Mozilla/4.73 [en] (WinNT; U)</HL> <HL>Referer: https://tokyo.itsc.austin.ibm.com:443/</HL> </HEADERS> </REQUEST> </HTTP-TRANSACTION> Figure D-3 Sample XML document
In todays technology, XML is starting to becoming a key piece of software infrastructure. The main idea is extremely simple. It is a language like HTML and is text based, but is rigidly enforced, and therefore can be built upon easily. XML documents may use a Document Type Definition (DTD) or an XML Schema. XML was designed to describe data and to focus on the data, unlike HTML, which was designed to display data. It was created to structure and store data. XML has these main applications: Sharing of information: The main problem integrating data between any two business organizations is the interface between them. If they can at least agree upon the standard of their common meeting point and its usage, they can build upon this to start building their applications. If there is already an existing interface or infrastructure provided by industry or government standard or infrastructure, the business cost of developing it is extinguished. Storage, transmission and interpretation of data: If the storage of information is adaptable to many mediums, its cost will be driven to the lowest
248
cost of all mediums. XML is based on text, which obviously is accepted by all. It can be read. Its storage is cheap, compared to the storage requirements of graphics. And because the size of this text based object is small, its transmission, not withstanding cost, is cheap as well. And because it is commonly accepted, since it adheres to world wide standards, it can be easily interpreted. Security: With the explosion of confidential information being transmitted across the Internet, there is now a need for sophisticated levels of security. Companies need to protect their documents and e-mail, banks need to allow their depositors to download their accounts, and merchants need to be available to their customers to enter their credit card details without compromising their security and privacy. The more secure the transmission medium, the more confidentiality it provides, the more it can be used to advertise a competitive advantage. With the evolution of XML digital signatures, the ability to protect or hide parts of a document, while sitting on a PC, server, or mainframe, now covers up a security patch. Speed and amount of content delivery: With the rapid evolution of network technologies, speed and delivery of any object has gained importance. Network companies now advertise download times of movies and CDs. Again, the first companies discovering the abilities of delivering something at an increasing speed without compromising their content will gain the competitive advantage. The working draft for XML 1.1 was published on the W3C Web pages in April 2002. XML 1.1 was formerly known as XML Blueberry. The XML 1.0 specifications were based on the Unicode Standard. However, the Unicode standards have evolved from Version 2.0 to 3.1 and beyond. XML 1.0 relied on the standard for character specifications. Characters that are not present in Version 2.0 would probably have be used in XML documents and character data. Developers would have developed workarounds for characters that were not supported in Unicode Version 2.0. These characters are not allowed in XML names, such as element type, names, and attribute names, just to name a few. Also, some characters should have been permitted in XML 1.0, but were not due to oversights and inconsistencies in Unicode 2.0. The philosophy for names has been reversed since XML 1.0. XML 1.1 names have been designed such that everything that is not forbidden is permitted. In XML 1.0, the philosophy was everything that was not permitted was forbidden. For example, under XML 1.0, if only a, b, c, d and e were allowed as names, that was all that could be used. In XML 1.1, we could say that we would not allow a and b. Therefore, we could use b, c, e, f, ...g, ...$, and so on. As Unicode grows past Version 3.1, changes to the XML can be avoided if nearly all characters are allowed. This will allow any kind of characters in a name.
249
A new version of XML is being created, rather than a set of errata, because the changes affect the definition of well-formed documents. XML Processors will recognize XML 1.1 documents from XML 1.0 documents by the declaration at the start of each document. For more details, visit the W3C Web site:
http://www.w3.org/TR/xml11/
Microsoft Excel for browsing XML files

One of the easiest way to browse XML log files from IBM Tivoli Monitoring for Network Performance is using Microsoft Excel. When you open an XML file with Microsoft Excel, it formats the XML document so that each XML field resides in a column. This is quite convenient for log files, as we can selectively hide unwanted columns or change its width or ever reorder the sort order. An IBM Tivoli Monitoring for Network Performance log file is not well formatted; in a sense, it needs to have an additional header and footer. We use a file called xmlheader, as shown in Example D-1.
Example: D-1 The xmlheader file <?xml version="1.0"?> <LOG>
We also use an xmlfooter file, which contain a single line:

</LOG>
We merge the files with the following command from a Windows workstation:
COPY xmlheader/b+msg_fnp_monitorc.log/b+xmlfooter msg1.xml
When we open msg1.xml using Microsoft Excel, we get a display similar to Figure D-4 on page 251.
250
Figure D-4 Sample XML log viewed using Microsoft Excel
Certificates and encryption

On unsecured networks like TCP/IP, there is concern for both the sender and the receiver about the security of the data that is sent over the network. The network protocol itself does not provide any protection against tampering with the data whatsoever. In order to transport sensitive data over the network, the following issues need to be resolved: Confidentiality: Only the parties involved in the data transfer should be able to read the contents of the data. Authentication: The parties should be able to identify each other beyond doubt. Data integrity: Data that has been altered during transmission should be detectable. Non-repudiation: The sender of the data should not be able to deny the data he sends.
251
The following sections discusses some approaches that provide resolution for these issues. More information on secure communication and infrastructure can be found in Deploying a Public Key Infrastructure, SG24-5512.
Secret Key Cryptography

The oldest cryptographic technique is known as Secret Key Cryptography. It is a way of achieving data confidentiality by encrypting and decrypting the data using a single key (the symmetric or secret key). Both the sender and the receiver of the data must have the secret key. In Figure D-5, A is sending a message to B, and both are using the same secret key to encrypt and decrypt the message.
A E nc ryp t with s e c re t ke y
C le artext m essag e
E ncryp ted m essag e
"Hello Alic e"
xc % s lk& ls kf
B D e c ryp t with s e c re t ke y
Trans fe r o f m e s s ag e
C lea rte xt m essag e
E ncryp ted m essag e
"Hello A lic e"
xc % s lk& ls kf
Figure D-5 Secret Key Cryptography
No one else can encrypt or decrypt the message, as they do not have access to the secret key. The advantage of Secret Key Cryptography is that it is fast. It is also less complex than Public Key Cryptography, which is discussed in Public Key Cryptography on page 253. The disadvantages of Secret Key Cryptography are: The distribution of the secret keys. When one party creates a secret key, how is the other party going to get the secret key? Send it in e-mail? That is insecure. Send it with normal mail? That can take a long time and is hard to automate.
252
Secret key administration. Another disadvantage is that for each party you want to communicate with, you must create a different secret key. If you do not, then the different parties would be able to read the communications of all the parties. You will end up with a lot of keys, which can become cumbersome to manage. Some well-known secret key algorithms are DES, Triple DES, Blow fish, IDEA, and RC5.
Public Key Cryptography

Public Key Cryptography is another widely used cryptographic technique that does not have the problems of multiple shared secret keys and the distribution of secret keys. The most well-known public key algorithm is RSA, but a technique called Elliptic Curves is becoming more widespread. Public Key Cryptography is a way of encrypting of data using an asymmetric key pair. One key, the Public key, is used for encrypting the data; the other key, the Private key, is used for decrypting the data. Both keys are different, and the key used for encrypting data cannot be used for decrypting the same data. The public key is made available to the world, while the private key should never be revealed. This technique is used for: Data encryption to provide confidentiality As an example, in Figure D-6 on page 254, A is sending a message to B. The message is encrypted with Bs public key, so that only B can decrypt the message using Bs private key.
253
A E n c ry p t w ith B 's p u b lic k e y
C le a rte xt m e s s a g e
E nc ryp te d m e s s a g e
"H e llo A lic e "
f m k lw # ls d k 0
B D e c ry p t w ith B 's p riv a te k e y
T ra n s f e r o f m e s s a g e
C le a rte xt m e s s a g e
E nc ryp te d m e s s a g e
"H e llo A lic e "
f m k lw # ls d k 0
Figure D-6 Public Key Cryptography
Data signing for authentication of the sender A, as the sender, wants to send data to B. A encrypts data with As private key, then B can decrypt the data with As public key. This way, B is sure that the data came from A, since no one else can create the message that can be decrypted with As public key. This process is shown in Figure D-7 on page 255. Note that although signing encrypts the data, it is not giving confidentiality, as anybody can decrypt the data by decrypting it with the senders public key. What signing provides is non-repudiation and data integrity (the sender cannot deny sending the message). The message cannot be changed, as that would make it impossible to decode it.
254
A S ign with A 's private key
Cleartext message
Signed message
"Hello Alice"
xc% slk&lskf
B unsign with A 's public key
Transfer of m essage
Cleartext message
Signed message
"Hello Alice"
xc% slk& lskf
Figure D-7 Signing of data with Public Key Cryptography
The combination of both processes can achieve confidentiality and non-repudiation. As shown in Figure D-8 on page 256, when A sends a message to B, the encryption/decryption process is as follows: 1. A signs the data with As private key. 2. A encrypts it with Bs public key. 3. When B receives the signed and encrypted data, B decrypts it using Bs private key 4. B checks the signature by unsigning (decrypting) it with As public key.
255
A Sign with A's private key Encrypt with B's public key
Encrypted message
Cleartext message
Signed message
"H Alice" ello
xc%slk&lskf
cdl$d)sl&f
Transfer of message
Cleartext message
Unsign with Decrypt with A's public key Signed message B's private key Encrypted message xc%slk&lskf cdl$d)sl&f
"Hello Alice"
Figure D-8 Signing and encryption using Public Key Cryptography
Public Key Cryptography has its own problems: The Public Key Cryptography process is computation intensive. Signing and encrypting the whole data stream implies a long encryption time. The difference in speed between Secret Key Cryptography and Public Key Cryptography is more than a factor of 1000. Man-in-the-middle attacks. Who guarantees us that the public key of Alice is really the public key of Alice? Somebody may publish a public key that is claimed to be someone elses public key and access communications encrypted with that fraudulent public key.
Refinements on cryptographic techniques

At the moment, we have a technique that is fast, but has problems from the management point of view, and a technique that has good manageability, but is slow, and has problems concerning authenticating the other party. Let us now discuss a number of techniques that can alleviate these problems.
Using hashes in Digital Signatures

We saw earlier that by encrypting a message with the private key of the sender, the message was signed, but also that this was a slow process due to the
256
inherent slowness of Public Key Cryptography. One way to overcome this problem is by creating a Message Digest through a hash function and subsequently sign the Message Digest. This process is shown in Figure D-9.
Cleartext m essage
M essage Digest H ash
Encrypt with private key
Digital signature
"H Alice" ello
wd$s*lsd^
cx3s&dsk)
Figure D-9 Generating a Digital Signature
A hash function is a tool that takes a message of any size and generates a small fixed size block of data (a Message Digest). The Message Digest has the following characteristics: The Message Digest is always the same for the same block of data. If the original message is altered by even one bit, the resulting digest of the changed message is very different. Message Digest creation through a hash function is very fast. It is impossible to generate the original message from the digest. So instead of signing the whole message and encrypting it afterwards (employing a Digital Signature), the encryption/decryption process shown in Figure D-10 on page 258 uses the following steps: 1. The sender, A, create a Message Digest. A encrypts the digest with As private key to create a Digital Signature. 2. The Digital Signature is appended to the original message, and the whole is encrypted with Bs public key. 3. The receiver, B, decrypts the whole message with Bs private key and separates the Digital Signature from the message. 4. B decrypts the Digital Signature with As public key to obtain the Message Digest. B also generates a Message Digest from the whole message with the same hash function. 5. B compares the two Message Digests. If both digests are the same, then B is sure who the message comes from and that the message is unchanged.
257
A
M essage Digest
Cleartext message
Sign with A's private key
Digital Signature
"H Alice" ello
hash Encrypt with B's public key
c4$dil
f6k#d
Cleartext message
Encrypted message & digest
"H Alice" ello
f6k#d
xc5kf&l'f
B
Unsign with A's public key
Digital Signature
Transfer of message
Message Digest
c4$dil
Compare Message Digest
f6k#d
Cleartext m essage
Decrypt with B's private key Encrypted message & digest xc5kf&l'f
c4$dil
hash
"H Alice" ello
Figure D-10 Public Key Cryptography using Digital Signatures
Some well known hash algorithms are SHA-1, MD5, and RIPEMD.
Digital Certificates
The problem of authentication of the public key can be solved by the use of Digital Certificates. A Digital Certificate binds the owner of the public key to the public key itself. It is a data structure that contains a public key, necessary details about its owner and some other information. All this information is signed by a trusted third party called a Certificate Authority (CA). Some important details about CAs are outlined below: 1. When a public/private key pair is generated, the public key, together with the identity of the owner, must be submitted to a CA. 2. The CA signs the data with his own private key. The data becomes a Digital Certificate, and returns it to the owner.
258
3. A certificate does not contain any confidential data, and should be made available to the world, so that other people can use this certificate for sending data to the owner of the certificate and decrypt data from the owner. There are many commercial CAs. Some examples, with their Web sites, are Verisign (http://www.verisign.com), Thawte (http://www.thawte.com), and Entrust (http://www.entrust.com). There are also some local CAs for each country. Commercial CAs certificates are often included in products like Web browsers. For Netscape, you can view the certificates of the different CAs (that it recognizes) by selecting Communicator Tools Security Info and clicking on the Signers link, as shown in Figure D-11.
Figure D-11 CAs Digital Certificates that Netscape recognizes
The use of the digital certificate in secure communication is shown in Figure D-12 on page 260.
259
A
compare Unsign challenge with B's public key
Transfer of message
B
Sign with secret key
Signed challenge
"message"
Signed challenge and B's certificate
"challenge"
xc%slk&lskf
"challenge"
xc%slk&lskf
Signed challenge and B's certificate
3
B's public key
xc%slk&lskf
B's certificate B's certificate
Unsign certificate with CA's public key
compare
Message Digest
Decrypt with B's private key
Hash
"message" signature A's certificate
4
we4^&4kls sldk$0-2+\
Encrypt whole message with B's public key
"message" A's certificate
we4^&4kls sldk$0-2+\
Unsign certificate with CA's public key
signature
A's public key
Unsign signature with A's public key
Message Digest
Figure D-12 Digital Certificates in Public Key Cryptography
1. A must obtain the certificate of B. If A cannot get it from a public directory, then A could send a challenge (a piece of arbitrary data) to the perceived address of B. 2. B signs the challenge using its private key, attaches Bs certificate to the challenge and sends it back to A. 3. A unsigns the certificate using a CAs public key, extract Bs public key and unsigns the challenge. If the original challenge is received, then A is certain about the identity of B. 4. A then generates a message with a digital signature, attaches As Digital Certificate and encrypts the whole with Bs public key. The result is sent to B. 5. After receiving the encrypted message, B decrypts it with Bs private key. The receiver then has access to the certificate of the sender, the message, and the signature of the message. a. B unsigns As certificate using the CAs public key and retrieve As public key.
260
b. B unsigns As signature using As public key and retrieves the Message Digest. c. B generates a digest from the original message and compares it with the unsigned message digest.
Adding Secret Key Cryptography to the mix

We now have established a secure way to transfer messages through Public Key Cryptography without having the authentication problems associated with it. The problem that remains is performance. It would be nice to marry the performance of Secret Key Cryptography with the management scalability of Public Key Cryptography. Here is how it works: 1. When two entities want to communicate, they set up communication channels using Public Key Cryptography. 2. During this set up, a secret key is created and transferred to the parties involved. This secret key will only be used for this session and will be destroyed afterwards. 3. Once all parties have the secret key, they can send information to each other with Secret Key Cryptography. 4. When the communication channel is closed, the secret key is destroyed.
Secure Sockets Layer protocol

In this section, we take a closer look on how the Secure Sockets Layer (SSL) protocol works. This protocol is used by the Tivoli Web Services Manager components to communicate securely over untrusted networks. SSL was conceived by Netscape Communications. It became the de facto standard to authenticate and encrypt communication between client and servers on TCP/IP networks. SSL performs the following functions: It authenticates the server to the client. Optionally, it authenticates the client to the server. It creates an encrypted connection between both machines. The authentication of the server to the client and vice versa happens through the exchange of certificates. The Certificate Authority that signed the certificate can be a different CA for the server than for the client. They must be trusted by the client and the server respectively.
261
The encryption of the connection allows you to keep the data sent over the connection confidential. On top of that, it also checks whether the data has been changed during the transfer. All the above functions of SSL depends strongly on the cryptographic principles described in Certificates and encryption on page 251. SSL sits between the TCP/IP protocols, who are responsible for the transport and routing of data over the Internet, and the application protocols, like Hypertext Transport Protocol (HTTP) or other application protocols. This is shown in Figure D-13. It consists of two protocol levels: Record layer protocol Communication protocols: Handshake protocol Change Cipher Specification protocol Alert protocol Application protocol
HTTP
LD AP
IM A P
SSL
H andshake P ro to c o l
C h a n g e C ip h e r P ro to c o l
A p p lic a tio n P ro to c o l
A le r t P ro to c o l
R e c o r d L a y e r P r o to c o l TCP IP
Figure D-13 SSL structure and place in the protocol stack
262
The Record Layer

All messages coming from the higher level protocols go through the Record Layer before going to the transport layer. The Record Layer sends blocks of data called records, which are of fixed length. A record contains the content type, the protocol version number, the length and the data, which is compressed and encrypted. Each message passes the following three functions: Fragmentation of data: The message is divided or combined to fit into a record. Remember that a record has a fixed length. Compression before sending the data. Encryption of the data part of the record.
The communication protocols

There are four communication protocols: The Handshake protocol defines the sequence of events to establish an SSL session between two entities. The Change Cipher Specification protocol is actually a subset of the handshake protocol. Its primary function is to indicate to the other party that there has been a change in the cryptographic options. The Alert protocol deals with errors. An alert message contains two parts: the actual error description and the severity level of the error. There are two levels of errors: Warning: This indicates a potential problem. An example is the close_notify error, which specifies that the sender will not send any more messages in the current session. Fatal: This interrupts the current session, and also means that the current session cannot be resumed in the future. An example of this is the bad_record_mac error, which indicates that the message or its hash has been tampered with. The Application protocol is responsible for passing messages from the application layer protocol to the record layer protocol. SSL communication is set up using the handshake protocol. Both entities negotiate the version of the protocol to be used (2.0 or 3.0), the cryptographic algorithms, and the setup of the keys. It is also possible to include entity authentication in this step (one-way or mutual). If this happens, the server will authenticate itself to the client using Public Key Cryptography, after which Secret Key Cryptography is used for performance, as discussed in Adding Secret Key Cryptography to the mix on page 261. The sequence of messages exchanged during the handshaking is illustrated in Figure D-14 on page 264.
263
C lie n t
S e rv e r
Request
1st C lient P hase P rotoc ol version S ession ID C ry ptographic options C ompression methods 1st S er ver P hase Certific ate S erver K ey E xc hange C ertific ate Request S erver Hello D one 2n d Client P hase C ertific ate C lient K ey E xc hange C ertific ate V erify c hangec ipherspec Finished
2nd S er ver P hase c hangec ipherspec Finished
Applic ation D ata
A pplic ation Data
Figure D-14 SSL 3.0 handshake protocol
1. A request is sent by the server to start the handshake process. This implies that the client has instigated the connection by requesting the appropriate URL from the server. 2. The first client phase reply contains: a. A protocol version. b. A random number, used for the generation of the session keys. c. A session ID, to check whether a new session needs to be set up. d. A list of cryptographic options: Key exchange algorithm, hash algorithm, and so on. e. A list of the compression methods supported by the client. After sending this message, the client waits for the first server phase. If it receives any other type of message, this results in a fatal error, and the handshake must be restarted. 3. During the first server phase, the server sends the following messages: a. It acknowledges which version of the SSL protocol is supported (lower than or equal to the version of the client). The server also generates a random number and returns the session ID from the first client phase, if it agrees to pick up the old session. A choice is also made from the set of cryptographic options and compression methods that were proposed by the client.
264
b. If the server authenticates to the client, the certificate of its public key is included. c. With the Server Key Exchange, the key exchange algorithm is specified. It can either be Diffie-Hellmann, RSA, or Fortezza. d. If the server wants the client to authenticate (mutual authentication), it sends a Certificate Request. 4. The client responds with the second client phase: a. If the server requested it, it sends its certificate. b. With the Certificate Verify, the client proves to the server that it is in possession of the private key that corresponds to the public key in its certificate by digitally signing a specific message (called a challenge). See Digital Certificates on page 258 for more details on Digital Certificates. c. The Client Key Exchange contains all the necessary information that both parties need to calculate all the session keys (the ephemeral secret keys). See Adding Secret Key Cryptography to the mix on page 261 for more details on this step. 5. Both parties now send, in turn, the finished message, preceded by the changecipherspec message, as shown in Figure D-14 on page 264. These are the first messages that use the newly agreed upon key material. SSL is a session based protocol and not a connection based protocol. A connection is set up every time there is data to be transferred between the client and the server. It is not necessary to go through the handshake protocol every time a new connection is made. Therefore, if previously arranged keys and algorithms are used, the session is resumed.
265
266
Abbreviations and acronyms

AIX APF API ASID AUTOTBL BPOOL CCITT Advanced Interactive eXecutive Authorized Program Facility Application Programming Interface Address Space Identifier Automation Table Bufferpool Comite Consultatif International Telegraphique et Telephonique (now ITU-T) Compact Disk Read Only Memory Central Data Warehouse Customer Information Control Systems Command Line Interface Central Processing Unit Communication Server for OS/390 Common System Area Communication Storage Manager Database 2 Data Encryption Standard Data Facility Data Set Services Document Type Definition Event Automation Services Extended Common System Area Event Integration Facility Enterprise Java Bean Extract, Transform, Load J2EE JCL JDBC JDK JES2 JFS JMS JMX JVM FTP HFS HPR HTML HTTP HTTPS I/O IBM ICMP IIS IMS ISMP ISO ISPF ITSM ITSO ITU-T File Transfer Protocol Hierarchical File Systems High Performance Routing HyperText Markup Language HyperText Transport Protocol HTTP Secure Input Output International Business Machines Corp. Internet Control Message Protocol Internet Information Server Information Management Systems Install Shield Multi Platform International Standard Organization Interactive System Productivity Facility IBM Tivoli Storage Manager International Technical Support Organization International Telecommunication Union-Telephony division Java 2 Enterprise Edition Job Control Language Java Database Connectivity Java Development Kit Job Entry Subsystem 2 Journaled File Systems Java Messaging Services Java Management eXtension Java Virtual Machine
CD-ROM CDW CICS CLI CPU CS390 CSA CSM DB2 DES DFDSS DTD EAS ECSA EIF EJB ETL
267
LDAP LRU LTPA MAT MD5 MIB MSU MTU MVS NLDM NMVT NPDA NPM ODBC OID OSA OSASF OSI OSPF PID PPI RACF RDBM RDBMS REXX RIM RMF
Light-weight Directory Access Protocol Least Recently Used Light-weight Third Party Authentication Message Automation Table Message Digest 5 Management Information Base Management Service Unit Message Transport Unit Multiple Virtual Storage Network Logical Data Manager Network Management Vector Table Network Problem Determination Assistant NetView Performance Manager Open Database Connectivity Object Identifier Open Systems Adapter OSA Service Functions Open Systems Interconnection Open Shortest Path First Process ID Program to Program Interface Resource Access Control Facility Relational Database Manager Relational Database Management Systems Restructured Executive eXtended eXecutor RDBMS Interface Module Resource Measurement Facility
RPC RRS SAX SDBM SDSF SGML SMP/E SNA SNMP SPUFI SQL SSL STC TBSM TCP/IP TDBM TDW TEC TSM TSO UDP URL USS VSAM VTAM WAR WEP
Remote Procedure Call Resource Recovery Services Simple API for XML Security Database Manager System Display and Search Facility Simple Generalized Markup Language System Modification Program Extended Systems Network Architecture Simple Network Management Protocol SQL Processing Using File Input Structured Query Language Secured Socket Layer Started Task Tivoli Business Systems Manager Transmission Control Protocol / Internet Protocol Traditional Database Manager Tivoli Data Warehouse Tivoli Enterprise Console Tivoli Storage Manager Time Sharing Option User Datagram Protocol Universal Resource Locator UNIX Systems Services Virtual Storage Access Method Virtual Telecommunication Access Manager Web Archive Warehouse Enablement Pack
268
WLM XCF XML ZFS
Workload Manager Cross-System Coupling Facility eXtensible Markup Language z Filesystem
Abbreviations and acronyms
269
270
Related publications
The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.
IBM Redbooks
For information on ordering these publications, see How to get IBM Redbooks on page 273. Note that some of the documents referenced here may be available in softcopy only. Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 4: Connectivity and Routing, SG24-6516 Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 5: Availability, Scalability, and Performance, SG24-6517 Communication Server for z/OS V1R2 TCP/IP Implementation Guide Volume 6: Policy and Network Management, SG24-6839 Deploying a Public Key Infrastructure, SG24-5512 OSA-Express Implementation Guide, SG24-5948 OSA-2 Implementation Guide (Update), SG24-4770 Useful TCP/IP and FTP Commands, TIPS0091
Other publications
These publications are also relevant as further information sources: IBM Tivoli Monitoring for Network Performance Administrator Guide, SC31-6364 IBM Tivoli Monitoring for Network Performance: Messages and Troubleshooting, SC31-6366 IBM Tivoli Monitoring for Network Performance Operator Guide, SC31-6365 IBM Tivoli Monitoring for Network Performance: Planning, Installation, and Configuration, SC31-6363 IBM Tivoli Monitoring for Network Performance Version 2.1 README File May 19, 2004, GI10-3255
271
IBM Tivoli Monitoring for Network Performance Version 2.1 Warehouse Enablement Pack, Version 1.1.0.0 Implementation Guide for Tivoli Data Warehouse, Version 1.2, SC31-6793 Implementing Tivoli Data Warehouse V1.2, SG24-7100 Open Systems Adapter-Express Customers Guide and Reference, SA22-7935 and SA22-7476 SNA Formats, GA27-3136 Tivoli NetView for z/OS Installation: Configuring Additional Components Version 5 Release 1, SC31-8874 z/OS Communications Server CSM Guide Version 1 Release 2, SC31-8808 z/OS V1R4.0 Security Server: LDAP Server Administration and Use, SC24-5923 z/OS V1R6 Communications Server: IP Configuration Guide, SC31-8775 z/OS V1R6 Communications Server: IP Configuration Reference, SC31-8776
Online resources
These Web sites and URLs are also relevant as further information sources: Entrust, Inc.
http://www.entrust.com
Free LDAP browser

http://www.iit.edu/~gawojar/ldap
Freeware for graphic toolkit

http://www.eteks.com/pja/en/#Download
Thawte, Inc.
http://www.thawte.com
Useful TCP/IP and FTP commands

http://www.redbooks.ibm.com/tips/TIPS0091/tips0091.pdf
VeriSign, Inc.
http://www.verisign.com
w3C: Extensible Markup Language (XML) 1.1

http://www.w3.org/TR/xml11
272
How to get IBM Redbooks

You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site:
ibm.com/redbooks
Help from IBM

IBM Support and downloads
ibm.com/support
IBM Global Services

ibm.com/services
Related publications
273
274
Index
Symbols
$CONFIG_DIR 25 $WAS_HOME 103 /etc/ibm/tivoli/common 126 /etc/ibm/tivoli/common/cfg 25 /etc/ibm/tivoli/common/cfg/log.properties 131 /etc/itmnp 128 /etc/itmnp/ 28 /etc/ldap 70 /etc/services 69 /home/db2inst1/sqllib/bin/db2start command 90 /home/db2inst1/sqllib/db2profile 90 /home/db2inst1/sqllib/java12 66 /opt/IBM/ITMNP21 67 /opt/IBM/ITMNP21/_uninst, 84 /opt/IBM/ITMNP21/bin/itmnp.jacl 83 /tmp 103 /u/itmnp 103104 /u/itmnp/jctest.ear 112 /u/itmnp/pja 109 /usr/lpp/db2/db8d/jcc/classes 107 /usr/lpp/itmnp/V2R1M0/bin/itmnpMonitor.sh 132 /usr/lpp/itmnp/V2R1M0/web_app_inst/itmnp21zos.tar 104 /usr/lpp/ldap/etc/ldap.profile 70 /usr/WebSphere 19 /usr/WebSphere/AppServer 69 /usr/WebSphere/AppServer/logs/server1 62 /var/ibm/tivoli/common/FNP 126 /var/ibm/tivoli/common/FNP/logs 31 automation specialist 56
B
backup command 95 backup schedule 96 baroc file 165 BEST_PRACTICES table 43 bind_interface 28
C
CA Call Level Interface, see CLI capacity planner 56 CCITT 242 CDW Central Data Warehouse, see CDW certificate authority 258 Certificate Authority, see CA CLI CLIENT 65 COLL_ATTR_GRP table 43 COLL_TBL table 43 COM.ibm.db2os390.sqlj.util.DB2DriverInfo.class 105 commands /home/db2inst1/sqllib/bin/db2start 90 backup 95 db2 91, 93 db2iupdt 65 db2start 94 db2stop 91 df 60 DIS BUFFERPOOL 107 DISPPI 176 extattr 131 itmnp_was.sh 90 itmnpMonitor.sh 132 kill 133 ldapcnf 71 mount 104, 121 NETMON 127 ovstatus 163 ps 133
A
access control list 247 Active Directory Service Interface 242 ADRDSSU 119 APF authorized attribute 131 API 242 asymmetric key pair 253 private key 253 public key 253 attribute 246 authType 65 automation blueprint 23
275
restore 98 startServer.sh 91 stopServer.sh 91 tar 92, 98 wrb 165 common directory 244 Communication port usage 35 Communications Storage Manager, see CSM console users 83 COPY utility 121 cryptographic principles 251 cryptographic technique public key cryptography 253 secret key cryptography 252 Crystal Enterprise 9 186, 209 Crystal Enterprise configuration 202 Crystal Enterprise Server 59 CSAPIport 28 CSM CSM_MON_MSMT table 43 CSM_SUMM_MSMT table 43
D
data source 85 data sources 26 database administrator 56 database configuration 93 DB2 10, 58 db2 command 91, 93 DB2 HFS code level 105 DB2 HFS directory 104 DB2 level 104 DB2 Universal Database Version 8 59 DB2Binder 107 DB2DriverInfo_native 104 db2iupdt command 65 db2j2classes.zip 105 db2Security 30 DB2SQLJJDBCPROGRAM 108 DB2SQLJPLANNAME 108 DB2SQLJSSID 108 db2start command 94 db2stop command 91 dbcache 25 DBCacheDirectory 25, 29 DBCacheRowLimit 30 DBCacheTimeout 29 DBDriverType 29
DBHostName 29 DBName 29 DBPassword 29 DBPort 29 DBUserName 29 df command 60 DFDSS 119 digital certificate 68 digital certificates 258 digital signature 257 Directory Access Protocol 243 Directory Information Tree 245 DIS BUFFERPOOL command 107 DISPPI command 176 distibuted consideration 51 distinguished name 245 distinguished name, see DN DN document organization 5 driver UNIVERSAL 29 DSNAQLDA 104 DSNL004I 29
E
EAS EE EE_AVL_MSMT table 44 EE_TT_DET_MSMT table 44 EE_TT_MSMT table 44 EIF enableCloudscape 30 Enterprise Extender, see EE ENUM_TYPES table 43 environment 5 ePortfolio 209 ETL ETL processing 197 event automation program 173 Event Automation Service, see EAS Event Integration Facility, see EIF event receiver 48 EVENT_DEST table 43 extattr command 131 extract, transform, and load, see ETL Extract-Transform-Load, see ETL
276
F
files /etc/services 69 /home/db2inst1/sqllib/db2profile, 90 /opt/IBM/ITMNP21/_uninst, 84 /u/itmnp 104 /u/itmnp/jctest.ear 112 /u/itmnp/pja 109 /usr/lpp/db2/db8d/jcc/classes 107 /usr/lpp/itmnp/V2R1M0/bin/itmnpMonitor.sh 132 /usr/lpp/ldap/etc/ldap.profile 70 /var/ibm/tivoli/common/FNP 126 fnp_config.xml 25 itmnp baroc 165 itmnp.jacl 83 itmnp.properties 25, 128 itmnp_install.log 83 itmnp_was.sh 90 itmnpServerKeyStore.jks 90 itmnpServerTrustStore.jks 90 log.properties 25 msg_fnp_monitor*.log 25 trace_fnp_monitor*.log 25 fnp_config.xml 25, 43 FNPI0002E 85 FNPlogPropertiesLocation 25 FNPM012E 131 FTP_CTRANS_MSMT table 44 FTP_SESS_MSMT table 44 FTP_STRANS_MSMT table 44 full image copy 121
HPR_PIPE_MSMT table 44 HPR_TT_MSMT table 44 HTTP HTTP secure, see HTTPS HTTPS Hyper Text Transfer Protocol, see HTTP
I
IBM automation blueprint 3 IBM automation platform 2 IBM Global Security Kit, see GSKIT IBM Tivoli Enterprise Console 16, 58, 161 IBM Tivoli Monitoring for Network Parformance startup 117 IBM Tivoli Monitoring for Network Performance 2 alert interface 161 architecture 7 automation sample 181 backup and receovery 91 backup and recovery 119 client certificates 39 communication 34 components 8 data collection 15, 141 database communication 40 database structure 41 discovery 161 environment 5 ETL processing 197 files 24 functions 8 graphic package 108 historical reporting 183 implementation components 48 implementation scenarios 47 introduction 1 monitor 22 monitor configuration 134 monitor installation 126 monitor problem determination 31 monitor start and stop 132 monitored metrics 138 monitoring best practices 137 native authentication 35 port usage 35 pre-defined reports 208 preparation 103 reinstallation 84
G
GENALERT Generic NMVT alert, see GENALERT graphic test program 112 GSKIT
H
hardware monitor 177 hash function 257 HFS Hierarchical File Systems, see HFS High Performance Routing, see HPR HPR HPR_AVL_MSMT table 44
Index
277
restore 98 roles 18 sample automation 173 sample reports 202 scope 5 security roles 77 shutdown 90, 117 SNMP data 26 startup 90 user authentication 34 verification 78, 115 Web application 10, 57 WEP installation 188 z/OS Web application 101 IBM Tivoli Monitoring for Network performance graph 16 IBM Tivoli NetView 58, 161 IBM Tivoli NetView for z/OS 16, 176 IBM Tivoli NetView for z/OS. 161 ICMP_RTT_MSMT table 44 IF_MULTI_MSMT table 44 IF_STATUS_MSMT table 44 IF_UNICAST_MSMT table 44 IHSAECFG 176 IHSAINIT 176 IIS implementation HFS files 103 implementation components 48 implementation roadmap 54 in a single collection 43 Install Shield Multi-Platform, see ISMP installation procedure 114 Install-Shield Multi Platform, see ISMP instanceName 65 Integrated TCP/IP Services Component, see ITSC Integrated TCPIP Services Components, see ITSC INTERFACE_OBJ table 42 Internet Information Server, see IIS INTERVL_TBL table 43 ISMP ITMAdmin 77 ITMNP 1 itmnp.baroc 165 itmnp.jacl 83 itmnp.properties 25, 129, 218 itmnp_install.log 83 itmnp_was.sh 90 itmnpds 85
itmnpItsc 162 itmnpMonitor.sh 132 itmnpServerKeyStore.jks 90 itmnpServerTrustStore.jks 90 ITMOperator 77 ITSC ItscNotify 85 ITU-T 242
J
J2C 36 Java Database Connectivity, see JDBC Java Development Kit, see JDK Java Management eXtension, see JMX Java Messaging Extension 36 Java Messaging Service, see JMS Java Messaging Services, see JMS Java Virtual Machine, see JVM jctest.ear 112 JDBC JDK JFS JMS JMSException 84 JMX Journal File System, see JFS Journaled File Systems, see JFS JVM
K
key generation 87 keyManagerPassword 30 keyStoreName 30 keyStorePassword 30 kill command 133
L
LAST_ETL_RUN table 43 LDAP LDAP user id 84 ldap.profile 224 ldapcnf command 71 LDAPSRV 72, 226 Lightweight Directory Access Protocol, see LDAP Light-weight Third Party Authentication, see LTPA listing applications 91 local_httpport 28
278
log files 31 log.properties 25 LTPA 87 LTPA key generation 87
M
mainframe data source 186 Management Information Base, see MIB MAT MAXASSIZE 128 MAXFILEPROC 128 MAXPROCSYS 128 MAXPROCUSER 128 MAXTHREADS 128 MAXTHREADTASKS 128 Message Automation Table, see MAT message digest 257 message.maxFiles 30 message.maxFileSize 31 metrics 138 MIB MIB_EXPR table 43 MIB_OBJ table 43 monitor files used 24 process structure 23 monitor configuration 134 monitor installation process 126 monitor.trace.level 31 monitor_cs390 process 28 monitor_hostname 28 MONITOR_INST_CFG table 43 mount command 104, 121 MQJMS1092 84 msg_fnp_monitor*.log 25
network system programmer 55 NETWORK_OBJ table 43 NMVT NODE_OBJ table 42 NPDA nvexportd 162
O
Object class 246 Object Identifier 247 offline backup 94 online backup 96 Open System Adapter, see OSA Open Systems Interconnection 243 OSA OSA Service Facility, see OSASF OSA_ETH_TT_MSMT table 44 OSA_STATUS_MSMT table 44 OSA_TT_MSMT table 44 OSA_TT_UTIL_MSMT table 44 OSASF OSNMPD process 127 ovstatus command 163
P
private key 253 ps command 133 public key 253 public key cryptography 253 confidentiality 253 data signing 254
R
RACF racfid 84 RDBM rearm value 140 REARM_GRP table 43 Redbooks Web site 273 Contact us xiv Relational Database Manager, see RDBM Relative distinguished name 245 Resource Access Control Facility, see RACF Resource Recovery Service, see RRS restore command 98 REXX client program 220 REXX server program 219
N
NETMON command 127 NetView alert automation 181 NetView Integrated TCP/IP services component 162 NETVIEW.SCNMUXCL 176 NETVIEW_ID_MAP table 43 network administrator 56 Network Management Vector Table, see NMVT network operator 56 Network Problem Determination Assistant, see NPDA
Index
279
role-based security 18 RRS rulebase 165 ruleset 165
S
SAF scenario comparison 50 schedules 140 Schema 247 scope 5 SDBM SDSNLOD2 dataset. 104 secret key cryptography 252 secure socket layer 261 alert protocol 262263 application protocol 262263 Change Cipher Specification protocol 262263 handshake protocol 262263 Record layer protocol 262 Secure Socket Layer, see SSL security administrator 56 Security Database Manager, see SDBM Security Database manager, see SDBM security issue authentication 251 confidentiality 251 data integrity 251 non-repudiation 251 SERVER 65 SERVER_ENCRYPT 65 service level coordinator 56 Simple Network Management Protocol, see SNMP SLAPDCNF 225 slapdcnf 71 slapdenv 71 SMARTSET_OBJ table 42 SMP/E SNA SNMP SNMP_EXPR_MSMT table 44 SNMP_INT_MSMT table 44 SNMP_STRING_MSMT table 44 socksPort 29 socksServer 29 SPUFI SQL program using file input, see SPUFI SSL
SSL problems 89 started task LDAPSRV 72 startServer.sh command 91 STK_IP_TT_MSMT table 44 STK_TCP_AVL_MSMT table 44 STK_TCP_TT_MSMT table 44 STK_UDP_TT_MSMT table 44 stopServer.sh command 91 symmetric key 252 System Authorization Facility, see SAF System Modification Program/Extended, see SMP/E Systems Network Architecture, see SNA
T
tar command 92, 98 TARGET_SET table 43 TCP/IP TCP/IP stack 142 TCP_APP_AVL_MSMT table 44 TCP_CONN_AVL_MSMT table 44 TCP_CONN_TT_MSMT table 44 TCP_PRV_MEM_MSMT table 44 TDBM TEC rulebase 58 rules 58 TEC rulebase 165 THRESH_GRP table 43 threshold value 140 Tivoli common directory 131 Tivoli Data Warehouse 4, 8, 183 Tivoli Desktop 165 Tivoli Enterprise Console, see TEC Tivoli Framework 165 Tivoli Storage Manager, see TSM TN_RTT_BKT_MSMT table 44 TN_RTT_BND_MSMT table 44 TN3270_AVL_MSMT table 44 TN3270_RESP_MSMT table 44 trace.maxFiles 30 trace.maxFileSize 30 trace_fnp_monitor*.log 25 TRACERT_MSMT table 45 Traditional Database Manager, see TDBM Transmission Control Protocol/Internet Protocol, see TCP/IP
280
trustStoreName 30 trustStorePassword 30 TSM
U
UDP_EP_TT_MSMT table 45 UNIVERSAL 29 Unix System Services, see USS user interface 12 USS
W
Warehouse Enablement Pack, see WEP WAS_hostname 29 WAS_httpport 29 Web application 10 structure 10 Web application on AIX 58 Web site administrator 56 WebSphere Application Server 10, 58 WebSphere settings 111 WebSphereServletProtocol 30 WEP WLM Workload Manager, see WLM wrb command 165
X
X Windows 86 X.500 243 XCF communication links 157 XML Document Type Definition 247 SAX 247 XML 1.0 specifications 249 XML trace 31
Z
z/OS 2 z/OS consideration 52
Index
281
282
(0.5 spine) 0.475<->0.875 250 <-> 459 pages
Back cover
IBM Tivoli Monitoring for Network Performance V2.1

The Mainframe Network Management Solution
Managing TCP/IP network performance from z/OS Sample implementation scenarios Operational examples and tips
This IBM Redbook explains the new IBM Tivoli Monitoring for Network Performance Version 2.1. This version of IBM Tivoli Monitoring for Network Performance provides a complete redesign of the z/OS TCP/IP management tools that was started by the NetView Performance Monitor for TCP/IP. IBM Tivoli Monitoring for Network Performance provides a comprehensive TCP/IP stack monitoring for z/OS. It collects performance metrics from the z/OS Communication Servers system management interface, measuring response time and Simple Network Management Protocol (SNMP) Management Information Base (MIB) variable collection. IBM Tivoli Monitoring for Network Performance uses strategic IBM software platforms, such as WebSphere Application Server, as the Web application platform, and DB2 Universal Database as the central repository. This redbook starts with exploring the architecture of IBM Tivoli Monitoring for Network Performance and its components. We also discuss various implementation scenarios and evaluate the benefits and drawbacks of each scenario. Implementation planning and consideration is presented and operational consideration is explained.
INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION
BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.
For more information: ibm.com/redbooks

SG24-6360-00 ISBN 0738491446

IBM Tivoli Monitoring For Network Performance V2.1 The Mainframe Network Management Solution Sg246360

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

IBM Tivoli Monitoring For Network Performance V2.1 The Mainframe Network Management Solution Sg246360

Hochgeladen von

Copyright:

Verfügbare Formate

Front cover

IBM Tivoli Monitoring for Network Performance V2.1

International Technical Support Organization

Copyright IBM Corp. 2004. All rights reserved.

Copyright IBM Corp. 2004. All rights reserved.

The team that wrote this redbook

Copyright IBM Corp. 2004. All rights reserved.

Figure 1 The redbook team

Become a published author

Send your comments in an Internet note to:

Introduction to network performance monitoring

Copyright IBM Corp. 2004. All rights reserved.

1.2 The automation blueprint

Business Service Management

Policy Based Orchestration

Figure 1-1 IBM automation blueprint

Chapter 1. Introduction to network performance monitoring

1.3 IBM Tivoli Monitoring for Network Performance

1.4 Redbook environment and scope

1.5 Document organization

Chapter 1. Introduction to network performance monitoring

Components and architecture

Copyright IBM Corp. 2004. All rights reserved.

2.1 Components and functions

NetView Integrated TCP/IP Services Component Windows/AIX

IBM Tivoli Monitoring for Network Performance monitor z/OS

Tivoli Data Warehouse Server Crystal Enterprise Server

Central Data Warehouse and Data marts

IBM Tivoli Enterprise Console IBM Tivoli NetView for z/OS

Figure 2-1 IBM Tivoli Monitoring for Network Performance components

Chapter 2. Components and architecture

2.2 Web application

2.2.1 Web application structure

itmnp21 Enterprise Application

ITMNPDB ITMNP JMX ITMNP EJB

Netview ITSC nvexportd

Figure 2-2 The Web application component structure

Chapter 2. Components and architecture

2.2.2 Web application user interface functions

Figure 2-3 Initial login window

Figure 2-4 Portfolio menu for an administrator

Chapter 2. Components and architecture

Chapter 2. Components and architecture

Figure 2-5 Table view of performance data

Figure 2-6 Graphical view

2.2.3 Role-based security

Chapter 2. Components and architecture

Figure 2-7 Role assignment for the Web application

2.2.4 Problem determination for Web application

Chapter 2. Components and architecture

Figure 2-8 Trace menu

Figure 2-9 WebSphere trace setting

Chapter 2. Components and architecture

Figure 2-10 Setting component trace

2.3 Monitor functions

2.3.1 Process structure of the monitor

itm np M o n ito r.sh

Java V irtu al M ach in e

Figure 2-11 Monitor startup process

Chapter 2. Components and architecture

2.3.2 Files used by the monitor

Figure 2-12 The monitor file usage