Information Technology: WFM - Run-Book

Information Technology
WFM – Run-book
[WFM]
Revision #: [1]
Created Date:7/18/2011
Last Revised:
Requested By: [VNOC Support]

Author: [Rajnish Tahiliani]
Error! Unknown document property name.Error! Unknown document property name.Error!
Unknown document property name.[Project Name] Page 2 of 28
Runbook Version #: [ ]
Table of Contents
Table of Contents ......................................................................................... 2
1. OVERVIEW ......................................................................................... 4
2. ROLES AND RESPONSIBILITIES ......................................................... 4
3. ESCALATIONS .................................................................................... 4
3.1 Severity Level Definitions ................................................................ 4

3.1.1 Severity 1 ................................................................................ 4
3.1.2 Severity 2 ................................................................................ 4
3.2 Escalation Procedures ...................................................................... 5

3.2.1 Time Frames ............................................................................ 6
3.3 Call Tree ........................................................................................ 6
3.3.1 User Call’s VMSD for a WFM Problem ........................................... 6
3.3.2 VNOC Gets a Monitoring Alarm ................................................... 6
3.4 GSS FAST Team Contact Info: .......................................................... 6
3.5 Infrastructure & Interfaces Team Contact Info: .................................. 7
3.6 How to Contact IEX Support ............................................................. 8

3.7 IEX Support Escalations: ................................................................. 9
4. ARCHITECTURE ................................................................................ 11
4.1 Architecture Diagrams ................................................................... 11

4.2 Dependencies ............................................................................... 13
5. HOST INFORMATION ........................................................................ 13
5.1 Infrastructure ............................................................................... 13

5.1.1 Network Diagram .................................................................... 13
5.1.2 IP Configuration Information .................................................... 13
5.1.3 Host Information .................................................................... 13
6. OPERATIONAL PROCEDURES ............................................................ 14
6.1 Admin Procedures and Maintenance – GSS WFM DB .......................... 20
6.2 Backup and Recovery – GSS WFM DB.............................................. 21
6.3 Troubleshooting Scenarios - Informatica .......................................... 21

7. FUNCTIONAL PROCEDURES - Informatica ........................................ 21
7.1.1 Functional Support Scenarios ................................................... 21
7.1.1.1 Running WFM Process ................................................................................21
7.1.2 SOX Evidence ......................................................................... 25
7.1.3 Emergencies .......................................................................... 25
7.1.4 Exceptions ............................................................................. 25
7.2 Failure Contingencies .................................................................... 25

8. SERVICE LEVEL AGREEMENT (SLA)................................................... 25
VMware CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error!
Unknown document property name.[Project Name] Page 3 of 28
9. MONITORING AND REPORTS ............................................................ 25
9.1 Types of Monitoring ...................................................................... 25
9.2 Monitoring Configurations .............................................................. 25
9.3 Weekly Reports ............................................................................ 25
9.4 Monthly Reports ........................................................................... 25
9.5 Ad-hoc Reports............................................................................. 25
9.6 Audit Logs ................................................................................... 25

10. Security and Compliance .................................................................. 25
10.1 Change Control ............................................................................ 25
10.2 SDLC .......................................................................................... 25
10.3 Known Deficiencies ....................................................................... 25
10.4 User Management ......................................................................... 25
10.5 Links to Access Control .................................................................. 25

10.6 SOX Evidence ............................................................................... 25
Appendix A: Document Change History ...................................................... 26
Appendix B: Open and Closed Issues ......................................................... 26
Appendix C: Glossary/ Definitions .............................................................. 26
Appendix D: Review and Sign-off ............................................................... 27
VMware CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document
property name.[Project Name] Page 4 of 28
1. OVERVIEW
The GSS Business team must effectively manage and track the time of its customer-facing
agents, which include Technical Support Engineers (TSEs), Customer Service Representatives
(CSRs) and Licensing Representatives. In the long-term VMware may consider expanding the
user base to external contractors and service providers.
With a global operations footprint, it has become critical to be able to optimally manage available
capacity to match customer demand. VMware GSS currently has very limited capabilities for
forecasting and scheduling without significant overhead. Additionally, the organization lacks
mature resource forecasting for its short/long term needs and the ability to effectively manage
sudden peak demand. Rudimentary performance tracking and resource availability are done with
legacy tools like Excel.
2. ROLES AND RESPONSIBILITIES

Informatica Team: Resolve Informatica maintenance, performance, and troubleshooting on production
Informatica server and data flow from the informatica server to the Work-force management application
server.
Telecom Team: Responsible for providing support during a data flow issue from CMs to Work-force
management application server
VNOC Team: Response and escalation for Severity 1 and 2 issues for WFM Application
DBA Team: Resolve database maintenance, performance, and troubleshooting on production
Informatica server
Network Team: Resolve network maintenance, performance, and troubleshooting on production
Informatica server
Linux Server Team: Resolve server maintenance, performance, and troubleshooting on production
Informatica server
Windows Server Team: Resolve server maintenance, performance, and troubleshooting on production
workforce management application servers
3. ESCALATIONS
3.1 Severity Level Definitions
3.1.1 Severity 1
HelpZilla: Critical issues affecting large numbers of people (e.g., entire campus, building, or regional
location). Response time: 30 minutes or less.
VNOC: LEVEL 1 Incident. Telecom System (CMS) is down.

Examples of LEVEL 1 Incident:
Gss team member are Gss team member will

Impacting entire site or
not getting RTA on WFM not be able to see SLA Severe business impact
greater
Application report
VMware CONFIDENTIAL
3.1.2 Severity 2
HelpZilla: Issues affecting an entire department (e.g., Engineering, Sales, Support, Finance, etc).
Response time: 1 hour or less.
VNOC: LEVEL 2 Incidents. Informatica is down or severely degraded.

Examples of LEVEL 2 Incidents:
Any of the above

systems are down or the Impacting certain Partial site functional
No workarounds
performance has department or group. loss.
degraded.
3.2 Escalation Procedures

Please note this escalation process is only if there is an infrastructure / operational level
outage within VMWare. All application level issues will be handled by the FAST GSS team. If users call
with a problem please contact FAST GSS Team.
Level 1 (Primary) Contact When to Notify Level 2

Alert Type
Person/Team Person/Team
WFM Server Issue ( Please notify FAST GSS Team
Please call IEX Support
Monitoring )
off-site (Prama B) or onsite (on-call Please notify FAST GSS Team
Informatica Availability
support resource)
Linux/Windows on-call appropriate to
CPU Threshold
the system.
Linux/Windows on-call appropriate to

Memory Threshold Please call FAST GSS Team
the system
If the server is Unix box and has a

Please call FAST GSS Team
database hosted on it, please call
Disk Capacity Linux/windows on-call appropriate to
DBA and Informatica admin. All
the system
other cases notify System Admin
Database Temp space Please call FAST GSS Team

DBA on-call
size monitoring

Database Availability DBA on-call

Listener Availability DBA on-call
VMware CONFIDENTIAL
3.2.1 Time Frames
Primary on-call person – please give 15 minutes to respond and call the secondary on-call person after the
15 minutes are over for the primary on-call person.
If there is no response from the primary/secondary on-call person, escalate it to the manager in the first
30 minutes.
3.3 Call Tree

GSS FAST Team will handle all application level issues. GSS FAST team will notify the users and also call
into IEX support as appropriate.
1. GSS End users (managers/supervisors, front line staff) would report application issues (such as with
forecasts, schedules, agent data corrections) with the FAST GSS team. The issues will be reported via
Help-SFDC-Zilla or can be sent directly to gss-global-wfmteam@vmware.com. Issues created in the SFDC-
Zilla are sent to this email address as well. The FAST GSS WFM team (Bhaskar, Karthik, Ashwin) and
the Futuresoft consultants subscribe to this email alias.
2. The FAST GSS WFM team (IEX secondary contacts) can directly report an issue with IEX that is related
to IEX application functionality or behavior. VMware issues can be tracked on IEX web portal using our
credentials.
Level 1 Support:
3.3.1 User Call’s VMSD for a WFM Problem

Contact the FAST GSS Team as per escalation process defined above. In case of an operations /
infrastructure issue please contact the Infrastructure & Interfaces Team Contact Info:
3.3.2 VNOC Gets a Monitoring Alarm

Please contact as per Escalations
3.4 GSS FAST Team Contact Info:
Escalation Path Name Office Phone Cell Phone
Primary GSS FAST

+91 966 387
During APAC Business Hours Bhaskar Babu None
4003
+91 959 114
“ Ashwin Prasad None
4998
+91 994 584
“ Karthik Venkatraman None
4255
During EMEA Business Hours John McDermott 353 21 4660290 353 86 7714575
During Americas Business Hours Jon Campbell 303 942 4867 303 601 7812
Manager Escalation Lynn Adamson 650 427-4317 415 794-4762
VMware CONFIDENTIAL
3.5 Infrastructure & Interfaces Team Contact Info:
Informatica Issue:
On-site Primary
Prama B
(Level 1 and 2)
Off-site Primary (Level 1 and 2) On-call Informatica resource
On-site/Off-site Manager
Rajesh Karale
(Level 1 and 2)
Telecom Issue:
Primary Hector Herrera 1-650-427-5528
Manager Escalation Will Clark 1-650-427-5775
Informatica - DNBIM Database Issue:

Primary On-call Oracle DBA resource
Manager Escalation Varinder Kumar (650) 427-5938 (408) 203-8602
Informatica - Oracle Apps Issue:

Primary On-call Oracle Apps resource
Manager Escalation Kiran Chinya (650) 427-5605 (408) 203-8103
Informatica - Oracle DB issue:

WFM DB issue:
Network issue:
VMware CONFIDENTIAL
Primary On-call Network matrix
Alan Holman
Manager Escalation
Will Stokes (650) 427-3231 (408) 316-7071
Business SMEs: SME’s in any S1/S2 issues related to their subject matter systems
SME IT BA Name IT BA Email
Telecom – Hector Herrera
Interfaces
VMStar – Deepa Shah
3.6 How to Contact IEX Support
Support Process 1 – Please click on image below for detailed Support Process
VMWare is provided with 24*7*365 Support Coverage with IEX.
VMware CONFIDENTIAL
Support Help Desk Line: 1.800.367.4390
Site # for Vmware 35420

Customer Interaction Center: http://cic.iex.com.
Please use site # as username
Password = “Virtualize”
techhelp@nice.com: Send documentation on an open incident. (This may include screen shots,
workstation traces or other pertinent information. Please reference the incident number within your
email.)
serial number wfm-35420
3.7 IEX Support Escalations:
Support Engineers and Help Desk Queue1.800.367.4390

IEX WFM Support Administrationtvadmin@nice.com
Engineers and Incident Documentationtechhelp@nice.com
Please follow this order for escalations
Jim Draucker
Manager, IEX WFM Support(O) 972.643.4617
Jim.draucker@nice.com(M) 214.934.9863
Scott McLaughlin
Manager, IEX WFM Support (O) 972.301.1783
scott.mclaughlin@nice.com(M) 972.983.4290
Carl Mullins
Manager, IEX WFM Support (O) 972.301.1781
carl.mullins@nice.com(M) 972.983.4297
Michael Darby
Director, IEX WFM Support, Americas(O) 972.301.1245
michael.darby@nice.com(M) 214.641.4827
Randy Price
Vice President, Customer Care(O) 972.301.4890
randy.price@nice.com(M) 214.563.8824
Customer Interaction Centerhttp://cic.iex.com

IEX Customer Services Websitehttp://totalview.iex.com
VMware CONFIDENTIAL
VMware CONFIDENTIAL
4. ARCHITECTURE
4.1 Architecture Diagrams
User User User

· Forecasts User
User User User User
· Schedules
User User User User User User User User
WFM Application Server
Smart Sync Smart Sync

Real time reports ( 30 Minute Interval ) Real time SR Information from MySQL
WFM Application Server WFM Application Server
CMS
MySQL
Avaya CM SFDC
WFM Database Server
The charts below show the end-to-end functionality of Informatica throughout all the Applications.
VMware CONFIDENTIAL
WFM Architecture Flow – Smart Sync SFDC
The following diagram depicts the end-to-end flow of data during the work flow management process and
the interaction of Informatica with mysql database and sending the file to the target location.
MYSQL
Staging tables
Informatica
Data File
VMware CONFIDENTIAL
4.2 Dependencies
WFM is dependent upon the following:
· WFM Application Server
· WFM DB Server
· Avaya CMS
· Informatica
o Oracle database (server and applications)
o Mysql
o it’s own host: the Informatica repository database
5. HOST INFORMATION
Host information for WFM PROD environment in SC9.
5.1 Infrastructure
5.1.1 Network Diagram
There is no network diagram.
5.1.2 IP Configuration Information
Informatica Server IP
ora-prod-inf-d2.vmware.com 10.113.61.44
WFM Application Servers IP
wfm-prod-app-1 10.113.61.232
wfm-prod-app-2 10.113.61.233
wfm-prod-app-3 10.113.61.234
WFM DB Servers DB/PORT
ora-prod-gsswfm-d1 WFMPROD/1521
CMS Server IP
Avaya CMS Server 10.19.69.130
5.1.3 Host Information

Ho
st
Description Server Location IP Address OS
Typ
e
inf-prod-app-
Informatica App SC9 10.113.61.44 Linux OEL5 VM
d2.vmware.com
Informatica (internally ora-prod-inf-

SC9 10.113.61.44 Linux OEL5 VM
dedicated) DB d2.vmware.com
WFM Application Server wfm-prod-app-1 10.113.61.232

SC9 Win2008 r2 VM
1
VMware CONFIDENTIAL
SC9 Win2008 r2 VM
2

SC9 Win2008 r2 VM
3
DB Server ora-prod-gsswfm-d1 SC9 Linux VM
6. OPERATIONAL PROCEDURES
Steps to Start WFM process – Informatica Interface:
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware CONFIDENTIAL
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to
following screen shot below
Username: (Credentials given to the User who runs the Loader)
Password: (Credentials given to the User who runs the Loader)
VMware CONFIDENTIAL
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
Note: The above credentials may not be the exact credentials given to the user who runs the Loader.
Step 4: infarep_PROD -> WFM-> Click on open.
VMware CONFIDENTIAL
Step 5: Click on Workflows -> wf_WFM_CASES -> Right click on Workflow and Click on Start Workflow
option.
VMware CONFIDENTIAL
The above step will kick off WFM job .
6.1.2 Steps to modify parameter value:
When there is any failure in Informatica job the jobs will be in unscheduled mode.
For ex: If the job is failed which is scheduled at 12.40 PM and its observed at 2 PM. Then you can
restart the job which will run for 12.00 to 12.30 interval.
But remaining missed 30 mins interval should be rerun manually updating the parameter table.
After that reschedule the jobs to run current time interval.
VMware CONFIDENTIAL
NOTE : The time interval should be in GMT format
Following are steps to update parameter value in Parameter file.
1) Login Informatica staging database. (SFA3PROD)

2) SELECT * FROM WFM_INFA_PARAMETERS ;
3) UPDATE WFM_INFA_PARAMETERS
SET PAR START_DATE=’ ‘ AND END_DATE=’ ‘ (Modify Parameter_Value)
4) Commit;
VMware CONFIDENTIAL
6.1 Admin Procedures and Maintenance – GSS WFM DB

Shutdown procedure
1. Login to ora-prod-gsswfm-d1 with your login

2. sudo to oracle
3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh)
4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD
5. Initiate sqlplus session
Sqlplus ‘/ as sysdba’;
6. Select name,open_mode from v$database < -- to make sure you are at correct
database
7. Shutdown Instance by typing
Shutdown immediate; < -- it make take 2-10 min depending on number of active sessions
in database
Startup procedure
1. Login to ora-prod-gsswfm-d1 with your login

2. sudo to oracle
3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh)
4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD
5. Initiate sqlplus session
Sqlplus ‘/ as sysdba’;
6. Startup
Select name,open_mode from v$database < -- to make sure it is correct database and
open_mode is read write.
Database Server Tns Entry

WFMPROD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-gsswfm-d1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = WFMPROD)
)
)
Catalog Server tns entry
PRODRCAT =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-rcat-d1.vmware.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = PRODRCAT.VMWARE.COM)
)
VMware CONFIDENTIAL
)
6.2 Backup and Recovery – GSS WFM DB

RMAN Backup:
The DB backup involves backing up of the 2 WFM Schema’s (WFMCUSTOMER & WFMYSTEM)
RMAN Backup job name : BACKUP_WFMPRD_ONLINE_DAILY scheduled daily at 11 PM
Export backup:
Export backup for wfmcustomer and wfmsystem user is scheduled via cron.
10 04 * * * /oracle/scripts/exp-wfmprod-user.sh 1>/oracle/scripts/log/exp-wfmprod-user.log 2>&1
6.3 Troubleshooting Scenarios - Informatica

Informatica Application Support troubleshooting scenario:
Scenario/Reported Condition Directives
Production Support person will receive Email
Notification and Session log attached with it to identify
Informatica workflow/session the issue.
failure Or
Open workflow monitor and open the corresponding
workflow/session logs to identify the issue
Open workflow monitor and open the corresponding
Informatica session did not start
workflow/session logs to identify the issue
7.FUNCTIONAL PROCEDURES - Informatica

7.1.1 Functional Support Scenarios
7.1.1.1 Running WFM Process
This process would be required to run every thirty minutes:
Drop the target files on Informatica server .

(path : /apps/infa/PowerCenter8.6.1/server/infa_shared/tgtFiles/WFM)
2.1 Steps to Start WFM:

VMware CONFIDENTIAL
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware CONFIDENTIAL
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to
following screen shot below
Username: (Credentials given to the User who runs the Loader)
Password: (Credentials given to the User who runs the Loader)
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
VMware CONFIDENTIAL
Note: The above credentials may not be the exact credentials given to the user who runs the Loader.
Step 4: infarep_PROD -> WFM-> Click on open.
1. Run the workflow under WFM folder - wf_WFM_CASES which will
VMware CONFIDENTIAL
Generate the output file. The output file is generated on the Informatica server at
/apps/infa/PowerCenter8.6.1/server/infa_shared/TgtFiles/WFM
And with the naming convention GMT format
Team Primary Contact Primary Contact Info Manager Name

IT –Informatica Ajit Deshpande
Meghana Kawari
Development
7.1.2 SOX Evidence

Not applicable
7.1.3 Emergencies
Contact appropriate tier-one resource (see call tree)
Route Helpzilla tickets to Informatica queue
7.1.4 Exceptions
Not applicable
7.2 Failure Contingencies

Contact appropriate tier-one resource (see call tree)
8. SERVICE LEVEL AGREEMENT (SLA)

9. MONITORING AND REPORTS
9.1 Types of Monitoring
9.2 Monitoring Configurations
9.3 Weekly Reports
9.4 Monthly Reports
9.5 Ad-hoc Reports
9.6 Audit Logs
10. Security and Compliance
10.1 Change Control
10.2 SDLC
10.3 Known Deficiencies
10.4 User Management
10.5 Links to Access Control
VMware CONFIDENTIAL
10.6 SOX Evidence
Not applicable
VMware CONFIDENTIAL
Appendix A: Document Change History

Version No. Date Name (Alias) Description of Change
1 7/3/2011 Tapash Jena Created the document
Appendix B: Open and Closed Issues

ID Issue Resolution Responsibility Target Date Impact
Date
Appendix C: Glossary/ Definitions

Term Definition
WFM Workflow management process
ETL Extract Transform Load
Infa Informatica
DB Database
VMware CONFIDENTIAL
Appendix D: Review and Sign-off

Below is a list of the required reviewers and approvers:
Name Role Contact Signoff Date
I have carefully assessed the Runbook Document for the (System Name). This document has been
completed in accordance with the requirements of the VMware System Development Life Cycle
methodology. I fully accept and signoff on this document.
NAME & Signature DATE

IT Operations (Director Level or above)
NAME & Signature DATE

IT Production Support (Director Level or above)
VMware CONFIDENTIAL

Information Technology: WFM - Run-Book

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Information Technology: WFM - Run-Book

Hochgeladen von

Copyright:

Verfügbare Formate

Information Technology

Requested By: [VNOC Support]

3.1 Severity Level Definitions ................................................................ 4

3.2 Escalation Procedures ...................................................................... 5

3.5 Infrastructure & Interfaces Team Contact Info: .................................. 7

3.6 How to Contact IEX Support ............................................................. 8

4.1 Architecture Diagrams ................................................................... 11

5.1 Infrastructure ............................................................................... 13

6.1 Admin Procedures and Maintenance – GSS WFM DB .......................... 20

6.2 Backup and Recovery – GSS WFM DB.............................................. 21

6.3 Troubleshooting Scenarios - Informatica .......................................... 21

7.2 Failure Contingencies .................................................................... 25

9. MONITORING AND REPORTS ............................................................ 25

9.1 Types of Monitoring ...................................................................... 25

9.2 Monitoring Configurations .............................................................. 25

9.3 Weekly Reports ............................................................................ 25

9.4 Monthly Reports ........................................................................... 25

9.5 Ad-hoc Reports............................................................................. 25

9.6 Audit Logs ................................................................................... 25

10.1 Change Control ............................................................................ 25

10.2 SDLC .......................................................................................... 25

10.3 Known Deficiencies ....................................................................... 25

10.4 User Management ......................................................................... 25

10.5 Links to Access Control .................................................................. 25

2. ROLES AND RESPONSIBILITIES

VNOC: LEVEL 1 Incident. Telecom System (CMS) is down.

Gss team member are Gss team member will

VNOC: LEVEL 2 Incidents. Informatica is down or severely degraded.

Any of the above

3.2 Escalation Procedures

Level 1 (Primary) Contact When to Notify Level 2

Linux/Windows on-call appropriate to

If the server is Unix box and has a

Database Temp space Please call FAST GSS Team

Please call FAST GSS Team

Please call FAST GSS Team

3.3 Call Tree

3.3.1 User Call’s VMSD for a WFM Problem

3.3.2 VNOC Gets a Monitoring Alarm

3.4 GSS FAST Team Contact Info:

Escalation Path Name Office Phone Cell Phone

Primary GSS FAST

Manager Escalation Lynn Adamson 650 427-4317 415 794-4762

3.5 Infrastructure & Interfaces Team Contact Info:

Primary Hector Herrera 1-650-427-5528

Manager Escalation Will Clark 1-650-427-5775

Informatica - DNBIM Database Issue:

Informatica - Oracle Apps Issue:

Informatica - Oracle DB issue:

3.6 How to Contact IEX Support

VMWare is provided with 24*7*365 Support Coverage with IEX.

Support Help Desk Line: 1.800.367.4390

Site # for Vmware 35420

3.7 IEX Support Escalations:

Support Engineers and Help Desk Queue1.800.367.4390

Please follow this order for escalations

Customer Interaction Centerhttp://cic.iex.com

User User User

WFM Application Server

Smart Sync Smart Sync

WFM Database Server

WFM Architecture Flow – Smart Sync SFDC

VMWare is provided with 247365 Support Coverage with IEX.