WLM z/OS v1r4

ATS-PSSC Montpellier Pascal Tillard Tillard@fr.ibm.com

23/06/2003

Benchmark Methodology

Workload Management
With WLM the service level administrator defines performance goals and assign a business importance to each goal. Goals are defined for work in business terms, and the system decides how much resource, such as CPU and storage, should be given to the work to meet its goal. Requires a shift of focus "from tuning at system resources level" "to defining performance expectations"
23/06/2003 / TP WLM & WAS 2

Workload Management
Performance Goals
Indicate the service level you would like to have for a given unit of work. There are three kinds of Goals:
Velocity Response Time Discretionary

23/06/2003 / TP

WLM & WAS

Workload Management
Service Class
A group of work with similar performance goals, business importance and resource requirements for management and reporting purposes

Workload
Aggregated set of Service Classes for reporting purposes

Reporting Class
Aggregate set of work for reporting purposes

23/06/2003 / TP

WLM & WAS

Workload Management
Resource Group
Processor capacity boundary across the sysplex. Assigns minimum and maximum amount of CPU service units / sec to work by assigning a Service Class to a Resource Group

Classification Rules - Classification Groups

Assign incoming work to a Service Class and Report Class Each Work Manager (ex. JES) has its own Classification definition

23/06/2003 / TP

WLM & WAS

Workload Management
Resource Group
Processor capacity boundary across the sysplex. Assigns minimum and maximum amount of CPU service units / sec to work by assigning a Service Class to a Resource Group

Classification Rules - Classification Groups

Assign incoming work to a Service Class and Report Class Each Work Manager (ex. JES) has its own Classification definition

23/06/2003 / TP

WLM & WAS

CB BATCH W O RKLO A D

Service Classes S ervice Classes

CB

Resource Group

BATCH

TRA N SACTIO N S JES CB CICS IM S TSO D B2 DDF ASCH OM VS IW EB LSFM

23/06/2003 / TP

Pd Goal im p Pd Goal im p Pd Goal im p

CB JES

Subsystem

Subsystem
Classification Rules & Groups

Report Class

page 8

WLM & WAS

Workload Management Services

Work Manager Services Execution Delay Monitoring Services Enclave Services Queuing Manager Services Routing Manager Services Sysplex Routing Services Scheduling Environment Services
WAS WAS5 WAS WAS

WAS

AS

23/06/2003 / TP

WLM & WAS

Work Manager Services

Work Manager Services allow WLM to:
Recognize a Subsystem Work Manager and the transactions it processes (IWMCONN - IWMDISC) Associate incoming work with a Service Class and optionally with a Report Class (IWMCLSFY) Recognize the address spaces that are processing the transactions

23/06/2003 / TP

WLM & WAS

Enclave and Application Environment

Functionalities that improve z/OS server capability
Enclave Services that allow management of a transaction across multiple address spaces and systems in a sysplex Application Environment that gives the ability to WLM to start additional server address spaces to meet the goals.

Application Environment are exploited by Work Managers using Queing and Routing Manager WLM services

23/06/2003 / TP

WLM & WAS

10

Enclave Services
An Enclave is an z/OS construct serving as:
unit of priority accounting for arriving transactions

Can spread across multiple dispatchable units and multiple address spaces Any number of Tasks and SRBs (preemptable) can be grouped together in an enclave Business objectives (goals) are derived from enclave associated service class, not from the server(s) service class where the enclave dispatchable units are executed Three Enclave types: Independent, Dependent, Foreign

23/06/2003 / TP

WLM & WAS

11

Enclave Resource Accounting

Enclave type Disp. Unit New Tran. Owner Srv.CLs. or Pgn CPU service
(preemtible time) (enclave tcb/srb)

Independent
SRBs / TCBs yes home a.s. iwmecrea work mngr. cls. rules smf30 owner sm72 enclave s.c.

Dependent
SRBs / TCBs no home a.s. iwmecrea owner smf30 owner smf72 owner s.c.

Foreign
SRBs / TCBs no original enclave owner smf30 origin. owner smf 97 exec. sys smf72 enclave s.c. excuting system(s) n/a n/a smf30 & smf72 server(s) a.s.

SRB service
(non-preemtible time)

n/a n/a smf30 & smf72 server(s) a.s.

n/a smf30 owner smf72 owner smf30 & smf72 server(s) a.s.

MSO service IOC service

23/06/2003 / TP

WLM & WAS

12

Queuing Manager Services

Subsystems using Queuing Managers services are logically divided in two major components:
Queuing Manager that receives transactions Server(s) that processes the transactions

The Queuing Manager:

receives work requests queues to WLM for the execution in server address spaces.

WLM dynamically starts and maintains server address spaces as required to meet the work manager's goals. Queuing Manager services:
use WLM Application Environment require a work manager to use Enclaves have a system scope
23/06/2003 / TP WLM & WAS 13

Queuing Manager - Server example

Queuing Manager AS Initialize subsystem and establish it as a queue manager Transaction starts
Accept work Create Enclave Queue Work Request IWMCONN
queue_manager = YES

WLM AS
Application Application Application Environment Environment Environment work queues IWMCLSFY IWMECREA IWMQINS
w/ Enclave Token

start
IWMCONN
= YES

Server Manager AS

Initialize and Connect to server_manager application environment Select Work from queue
Process Request Complete Delete Enclave

IWMSSEL
IWMSTBGN IWMSTEND IWMEDELE

SC A

SC B

Remove Subsystem and terminate

IWMSSEL
IWMDISC

Transaction Ends

RC=0C

Told to terminate
clean up, disconnect from application environment terminate

IWMDISC

23/06/2003 / TP

WLM & WAS

14

Application Environments
Server A.S. management function is implemented through the Application Environment concept A.E. is a grouping of transactions belonging to a Work Manager that use the same application libraries A.E.s can have System or Sysplex scope
Work Managers using WLM Queuing services have System scope (ex: WebSphere, DB2 Stored Procedure) Work Managers using WLM Routing services have Sysplex scope (ex: SOM)

A.E.s are defined using WLM policy Incoming requests should be assigned to an A.E. by the Work Manager and queued to WLM WLM manages Server A.S. (start/stop)

23/06/2003 / TP

WLM & WAS

15

WLM-Managed Initiators - Review

Individual batch queues can be WLM managed Initiators are dynamically started by WLM to meet service class period goals by reduction of batch queue delays and to optimize resource usage WLM selects the system based on
Available system resources Availability of waiting batch jobs Importance of the service class

Initiators are dynamically stopped by WLM

When significantly more initiators exist than needed (> 1.5 times avg. queue length) In case of CPU or memory shortage After one hour of inactivity
23/06/2003 / TP WLM & WAS 16

Limitations Before z/OS V1R4

Balancing only when new initiators are Started When initiators are already available, they will be used New jobs can be started on any system where idle initiators are, regardless of system load Often tends to batch jobs selected on heavily loaded systems, while other systems in the sysplex have relaxed capacity available Hurts discretionary workload on loaded systems

23/06/2003 / TP

WLM & WAS

17

Enhanced Batch Initiator Balancing

Balance distribution of initiators over sysplex members More aggressive reduction of initiators on heavily loaded systems (< 5% CPU capacity) when other systems have enough idle capacity Start of new initiators on systems with relaxed capacity (up to 5 initiators at once) Balancing evaluation done every 10 seconds WLM keeps enough initiators available for jobs with system affinity to constrained systems
23/06/2003 / TP WLM & WAS 18

WAS / WLM Interaction

W ebSphere Application Server Instance
SR 6 SR 6 SR 5 SR 5 SR 4 SR 4
1 6 5 4 3
2

6 Service C lasses:

S C =CB 05 S C=CB 05

SR 3 SR 3
2

SR2 SR2

SC =CB04 SC=C B04

CR
5 IIO P IIO P

SR 1 SR 1

S C =CB 03 S C=CB 03

SC=C B02 SC=C B02

H TTP H TTP

SC=IW EB01 SC=IW EB01

S C =CB 01 S C=CB 01

M IN _SR S=1 M AX_SR S>=6

23/06/2003 / TP

WLM & WAS

19

WLM - Classifying WebSphere Address Spaces

Daemon, Naming, SMS, Application Control Regions
Classify as SYSSTC or high velocity

Application Server regions

Classify with velocity goal, high enough to get started
Work is actually classified under the application environment

Garbage Collection runs under this service class.

Don't make it too high because GC is CPU & storage-intensive

23/06/2003 / TP

WLM & WAS

20

WLM - Classifying WebSphere Address Spaces

Sample STC Classification rules: (use Report Classes!)
Qualifier Qualifier Starting Service Report # type name position Class Class - ---------- -------------- --------- -------- -------1 TN WSLDAP OPS_HI RWSLDAP 1 TN WSAPP1C OPS_HI RWSAPP1C 1 TN WSAPP1S OPS_MED RWSAPP1S 1 TN WS%%%%C OPS_HI RWSCTLRG 1 TN WS%%%%S OPS_MED RWSSVRRG

OPS_HI service class:

Importance= 1, Velocity = 70

OPS_MED service class: Importance= 2, Velocity = 40

23/06/2003 / TP WLM & WAS 21

WLM - Classifying WebSphere Transactions

Subsystem type = CB using the following criteria:
Server name (CN) Server instance name (SI) - not useful because instances share work Userid assigned to the transaction (UI) - not useful unless RUNAS=USER Transaction class (TC) - assigned from env. vars or a "URI mapping file":
BBOC_HTTP_TRANSACTION_CLASS= (default for server or SI) BBOC_HTTP_SSL_TRANSACTION_CLASS= (same if coming in via SSL) BBOC_HTTPALL_TCLASS_FILE= (points to HFS file - see next foil)

Percentage response time goal is recommended

Ex. 80% of trans less than 0.5 seconds (or high velocity default SC) Default is SYSOTHER ("scum")
23/06/2003 / TP WLM & WAS 22

Classification by Virtual host, Port, or URI

Qualifier Qualifier # type name - --------- -------1 2 2 1 2 2 CN . TC . TC CN . UI . TC Start Service Report position Class Class --------------- -------Default: CBMED RCBDEFLT WSPROD 1 CBMED RWSPROD . TCLASS1 CBFAST RWSPRD1 . TCLASS2 CBMED RWSPRD2 WSTEST 1 CBSLOW RTSTEST . USER1 CBMED RTSTSTU2 . TCLASS5 CBSLOW RTSTST5

Transaction Class Mapping file format:

TransClassMap TransClassMap TransClassMap TransClassMap TransClassMap TransClassMap <host>:<port> www.ibm.com:80 www.ibm.com:* *:443 *:* * <uritemplate> /Webap1/myservlet /Webap1/myservlet * /Webap1/myservlet * <tclass> TCLASS1 TCLASS2 TCLASS3 TCLASS4 TCLASS5

Examples: ibm.com:80/Webap1/myservlet => TCLASS1 => CBFAST (RWSPRD1) ibm.com:443/Webap1/xservlet => TCLASS2 => CBMED (RWSPRD2)
23/06/2003 / TP WLM & WAS 23

Monitor I
REPORT CLASS=RWASPAS TRANSACTIONS AVG 1.00 MPL 1.00 ENDED 0 END/S 0.00 #SWAPS 0 EXCTD 0 AVG ENC 0.00 REM ENC 0.00 MS ENC 0.00 TRANS.-TIME SS.TTT ACTUAL 0 EXECUTION 0 QUEUED 0 R/S AFFINITY 0 INELIGIBLE 0 CONVERSION 0 STD DEV 0 DESCRIPTION =WAS Controller Region --SERVICE RATES-ABSRPTN 117 TRX SERV 117 TCB 4.7 SRB 0.4 RCT 0.0 IIT 0.0 HST 0.0 APPL % 1.7 PAGE-IN RATES SINGLE 0.0 BLOCK 0.0 SHARED 0.0 HSP 0.0 HSP MISS 0.0 EXP SNGL 0.0 EXP BLK 0.0 EXP SHR 0.0 ----STORAGE---AVG 4588.00 TOTAL 4587.99 CENTRAL 4535.99 EXPAND 52.00 SHARED 0.00

REPORT CLASS=RWASPASS TRANSACTIONS AVG 2.00 MPL 2.00 ENDED 0 END/S 0.00 #SWAPS 0 EXCTD 0 AVG ENC 0.00 REM ENC 0.00 MS ENC 0.00 TRANS.-TIME SS.TTT ACTUAL 0 EXECUTION 0 QUEUED 0 R/S AFFINITY 0 INELIGIBLE 0 CONVERSION 0 STD DEV 0

DESCRIPTION =WAS Server Region --SERVICE RATES-ABSRPTN 333 TRX SERV 333 TCB 2.6 SRB 0.0 RCT 0.0 IIT 0.0 HST 0.0 APPL % 0.9 PAGE-IN RATES SINGLE 0.0 BLOCK 0.0 SHARED 0.0 HSP 0.0 HSP MISS 0.0 EXP SNGL 0.0 EXP BLK 0.0 EXP SHR 0.0 ----STORAGE---AVG 85878.1 TOTAL 171756 CENTRAL 171756 EXPAND 0.00 SHARED 4365.99

REPORT CLASS=RWASWEB TRANSACTIONS AVG 0.21 MPL 0.21 ENDED 4974 END/S 16.58 #SWAPS 0 EXCTD 0 AVG ENC 0.21 REM ENC 0.00 MS ENC 0.00 TRANS.-TIME SS.TTT ACTUAL 12 EXECUTION 12 QUEUED 0 R/S AFFINITY 0 INELIGIBLE 0 CONVERSION 0 STD DEV 12

DESCRIPTION =Report WAS TCWASPAS Enclave --SERVICE RATES-ABSRPTN 1292 TRX SERV 1292 TCB 11.9 SRB 0.0 RCT 0.0 IIT 0.0 HST 0.0 APPL % 4.0 PAGE-IN RATES SINGLE 0.0 BLOCK 0.0 SHARED 0.0 HSP 0.0 HSP MISS 0.0 EXP SNGL 0.0 EXP BLK 0.0 EXP SHR 0.0 ----STORAGE---AVG 0.00 TOTAL 0.00 CENTRAL 0.00 EXPAND 0.00 SHARED 0.00

23/06/2003 / TP

WLM & WAS

24

Littles Law
N

: customer arrival rate N: average number of customers in system T: average delay per customer in system Littles Theorem: System in steady-state
N = T
23/06/2003 / TP WLM & WAS

Lq = W
25

Proof
Arrival Threads rate tx/s number 100 100 200 300 400 400 500 600 700 1 3 3 3 3 6 10 10 10 Ended/sec Time/tx TCB time Waiting MPL Appl% millisec /tx ms Time ms 7 2,8 4,2 28 99,36 0,7 6,8 2,7 4,1 99,43 0,68 27,6 7,4 2,8 4,6 199,33 1,48 55,3 9,1 2,8 6,3 299,38 2,73 85,9 17,2 2,7 14,5 396,29 6,82 109,3 8,7 2,8 5,9 399,66 3,48 112,1 11,4 2,8 8,6 499,36 5,7 142,2 30 2,9 27,1 598,45 17,99 176,3 40,6 2,8 37,8 696,79 28,31 200,9
Little's Law
resp.time ms 50 MPL 30 25 20 15 10 5 0

0 tx/sec resp.time ms MPL

100 6,8 0,68

200 7,4 1,48

300 9,1 2,73

400 17,2 6,82

500 11,4 5,7

600 30 17,99

700 40,6 28,31

23/06/2003 / TP

WLM & WAS

26

Enclave report limitation

Problem
Currently no possibility exists to report more than the response time for enclaves for subsystems/applications creating enclaves for transaction performance management. Subsystems like CICS and IMS have the ability to create monitoring environments and present subsystem states for reporting and analysis purposes.
23/06/2003 / TP WLM & WAS 27

Solution
External interfaces for monitoring environments have been changed to allow WLM to establish report-only performance blocks (PBs) External interfaces have been changed to enable association of reportonly PBs with enclaves and address spaces Extended sampling for report-only performance blocks Collect data for other goal types than just response time goals Include states for multi-period service classes

Exploiter
Websphere Application Server for z/OS (in a new WAS release) WLM support in R4, PTFs rolled back to R2 & R3
23/06/2003 / TP WLM & WAS 28

Execution Delay Monitoring Services

Allow WLM to know how well the work is executed and where delays are occurring Only response time goals can be used WLM does not individually manage transactions resource utilization WLM can allocate resources to address spaces based on the behavior of the transactions being executed by them Used by CICS , IMS and WAS v5

23/06/2003 / TP

WLM & WAS

29

Externals
IWMMCREA - Create Delay Monitoring Environment - changed IWMMDELE - Delete Delay Monitoring Environment IWMMINIT - Initialize Delay Monitoring Environment - changed IWMMRELA - Relate Delay Monitoring Environment - changed IWMMCHST - Change State of Work unit (inline only) - changed IWMMXFER - Monitoring Environment Transfer (inline only) IWMMSWCH - Monitoring Environment Switch IWMMNTFY - Notify of work execution completion - changed IWMRPT - Report Response Time - changed IWMMEXTR - Monitoring Environment Extract Service - changed IWMMABNL - Monitor Environment Abnormal Event Service (inline only) IWMMREG - Resource monitoring registration IWMMDREG - Resource monitoring deregistration

23/06/2003 / TP

WLM & WAS

30

Z/OS Worload Simulator Architecture

Windows Controller
Browser Stop simulation Start simulation Monitor run GUI application Capture Play back Analyse

z/OS Engine
config files

ftp
Web Monitor

Host AIX - WIN - z/OS - Linux

Performance agent

Http log

trace file

Http Server
Workload Simulator Daemon

WebSphere AS
http/https

Http error xml report

target application

23/06/2003 / TP

WLM & WAS

31

RMF Mon III: Response Time distribution

23/06/2003 / TP

WLM & WAS

32