BCAF 2013 in A Nutshell Study Guide For Exam 143-130: Brocade University Revision 0213

BCAF 2013 in a
Nutshell Study Guide

for Exam 143-130
Brocade University
Revision 0213
Corporate Headquarters - San Jose, CA USA
T: (408) 333-8000
info@brocade.com
European Headquarters - Geneva, Switzerland
T: +41 22 799 56 40
emea-info@brocade.com
Asia Pacific Headquarters - Singapore
T: +65-6538-4700
apac-info@brocade.com
2013 Brocade Communications Systems, Inc. All Rights Reserved.
ADX, AnyIO, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, ICX, MLX,
MyBrocade, OpenScript, VCS, VDX, and Vyatta are registered trademarks, and HyperEdge, The
Effortless Network, and The On-Demand Data Center are trademarks of Brocade
Communications Systems, Inc., in the United States and/or in other countries. Other brands,
products, or service names mentioned may be trademarks of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty,
expressed or implied, concerning any equipment, equipment feature, or service offered or to
be offered by Brocade. Brocade reserves the right to make changes to this document at any
time, without notice, and assumes no responsibility for its use. This informational document
describes features that may not be currently available. Contact a Brocade sales office for
information on feature and product availability. Export of technical data contained in this
document may require an export license from the United States government.
Revision 0213
2013 Brocade Communications Systems, Inc. All Rights Reserved. iii
Brocade Certified Architect for FICON 2013 in a Nutshell First Edition
Introduction to BCAF 2013 in a Nutshell First Edition
Objective: The BCAF 2013 Nutshell guide is designed to help you prepare for the BCAF 2013 Certification,
exam number 143-130.
Audience: The BCAF 2013 Nutshell self-study guide is intended for those who have successfully completed
the CAF200 Fundamentals of Brocade Mainframe Networking course, and who wish to undertake self-study
or review activities before taking the actual BCAF 2013 exam. The BCAF 2013 guide is not intended as a
substitute for classroom training or hands-on time with Brocade products.
How to make the most of the BCAF 2013 guide: The BCAF 2013 guide summarizes the key topics on the BCAF
2013 exam for you in an easy to use format. It is organized closely around the exam objectives. We suggest
this guide be used in conjunction with our free online knowledge assessment test. To benefit from the BCAF
2013 guide, we strongly recommend you have successfully completed the CAF200 Fundamentals of Brocade
Mainframe Networking course.
We hope you find this useful in your journey towards BCAF 2013 Certification, and we welcome your feedback
by sending an e-mail to jcannata@brocade.com.
Joe Cannata
Certification Manager
iv 2013 Brocade Communications Systems, Inc. All Rights Reserved.
2013 Brocade Communications Systems, Inc. All Rights Reserved. v
Brocade Certified Architect for FICON 2013 16Gbps in a Nutshell Second Edition
Table of Contents
1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
System z Host, I/O Channel Subsystem and FICON Protocol Architecture . . . . . . . . . . . . . . . 1
Mainframe Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Mainframe Models: zSeries to System z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
What is FICON? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
FICON Frame Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
FICON Operating Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
FCP and FICON Mode Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Channel Subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Multiple CSS Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Logical Channel Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Channel Spanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Open Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
FICON Connection Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
FICON Express Cards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Brocade MLX Series Multiservice Ethernet routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
z/OS Discovery and Auto-configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Parallel Sysplex and Coupling Links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
zHPF - High Performance FICON. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Brocade Network Advisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Event Notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Reasons to Migrate from ESCON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
FICON CUP Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Features of CUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Prohibit Dynamic Connectivity Mask (PDCM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Mainframe Reporting with RMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Creating RMF 74-7 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
FICON CUP and FOS Firmware Upgrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2 Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Single Director vs. Cascaded Director Design Requirements . . . . . . . . . . . . . . . . . . . . . . . . 21
2-byte Link Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Configuring CHPIDs Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Scalability Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Buffer-to-Buffer Credits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Long Distance Link Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
FICON Emulation Requirement for a Determinate Path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
FCIP Tunnel Between Sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Brocade FCIP Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
vi 2013 Brocade Communications
High Performance FCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Traffic Isolation Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
FC Long Distance Cascading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
FCIP Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Protocol Intermix Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
FICON and FCP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Virtual Fabrics and FICON. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Supported Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Cloud Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Inter-chassis Links (ICLs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
N_Port ID Virtualization (NPIV) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Benefits of Switched FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Minimize Mainframe Channel Card Costs with Switched FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Remote Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Synchronous Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Asynchronous Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
FICON Emulation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Bandwidth Allocation and Restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3 Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Zoning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Domain ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Switch ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Fill Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Traffic Isolation Zoning Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
TI Zone Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Illegal ETIZ Configuration: Separate Paths from a Port to Devices on Same Domain . . . . . . . . . . . . . . . . . . 42
256-area Addressing Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Port Swapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Geographically Dispersed Parallel Sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Troubleshooting Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Native Connectivity (M-EOS interoperability) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
ISL Decommission. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Command Line Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Diagnostic Port. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Pathinfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Blade Swapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Web Tools - Performance Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
SAN Health. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
IOCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Types of Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2013 Brocade Communications Systems, Inc. All Rights Reserved. vii
FICON CUP Zoning and PDCM Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Fabric Watch Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Reasons to Customize Fabric Watch Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Fabric Monitoring Setting Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Advanced Performance Monitoring Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
RMF 74-7 Records. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
RMF FICON Director Activity Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
IFCCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Saving or Copying Allow/Prohibit Matrix Configurations to Another Device . . . . . . . . . . . . . 52
Cascaded FICON fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Firmware Downloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
FCIP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5 Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Top Talker Monitors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
General Information to Gather for all Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Taking the Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
viii 2013 Brocade Communications
2013 Brocade Communications Systems, Inc. All Rights Reserved. ix
List of Figures
IBM System z10 Mainframe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
I/O definition comparison (FCP to FICON and ESCON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Channel Subsystem Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
FICON Connection Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Parallel Sysplex connectivity with mixed Coordinated Timing Network (CTN) . . . . . . . . . . . . . . . . . . . . . . . .11
IBM and Brocade Integrated TS7700 Grid Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
CUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Single vs. Cascaded FICON Director Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
2-Byte Link Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24
Sample B-Series Trunk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
FCIP tunnel and circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
FC long distance cascading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Brocade DCX with ICLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Illegal ETIZ configuration: two paths from one port to two devices on the same remote domain . . . . . . .42
SAN Health Output with IOCP Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47
FICON Director Activity Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50
FICON Director Activity Report for an M6140 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51
x 2013 Brocade Communications Systems, Inc. All Rights Reserved.
2013 Brocade Communications Systems, Inc. All Rights Reserved. xi
List of Tables
Brocade enterprise-class platform blade terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
xii 2013 Brocade Communications Systems, Inc. All Rights Reserved.
2013 Brocade Communications Systems, Inc. All Rights Reserved. 1
1 Architecture
After completing this section you should be able to read a scenario and demonstrate your knowledge of the
following:
Describe System z host, I/O channel subsystem, and FICON protocol architecture.
Demonstrate knowledge of System z channel I/O system configuration.
Describe System z I/O, channel, and CUP interdependencies.
System z Host, I/O Channel Subsystem and FICON Protocol Architecture
Mainframe Overview
A mainframe is one machine that is a central electronics complex (CEC)
A CEC will have many Central Processing Units (CPUs) in it
A CEC can support logical partitions (LPARs)
System z can support up to 60 LPARs
CPUs are allocated to LPARs dynamically
Special CPUs are called Service Assist Processors (SAPs)
Each LPAR can run a different Operating System such as z/OS and Linux
Mainframe is simply an industry term for a large computer. The majority of mainframes today are
manufactured by IBM. The mainframe has historically been associated with centralized rather than
distributed computing but IBM refers to its mainframes as a large server. These mainframes can be used to
serve distributed users and smaller servers in a computing environment. This is made possible with
capabilities of these to run thousands of virtual Linux images connected via Fibre Channel to storage.
Mainframe Models: zSeries to System z
In 2000, IBM launched the zSeries 900 line of servers which were referred to as the zSeries and included
models 800, 890, 900, and 990
In 2000, IBM renamed the System/390 to IBM eServer zSeries
In 2005, IBM launched the z9 series which included IBM z9 Enterprise Class and Business Class
In 2008, the z10 Enterprise Class was launched
Since FICON has been available since the late 1990s, the 9672 G5/G6 was the first mainframe to provide
support for FICON
2 2013 Brocade Communications Systems, Inc. All Rights Reserved.
zSeries, System z including z9 and z10
- 800, 890 (midrange enterprise)
- 900, 990 (enterprise, large enterprise; business class UNIX)
- z9, z10 (enterprise, large enterprise; business class UNIX)
- z Operating System (z/OS)
- z Virtual Machines (z/VM)
- z Transaction Processing Facility (zTPF)
- Linux for System z
Elements that affect unrepeated distance
- The type of fiber used
- The port transceiver type
FIGURE 1 IBM System z10 Mainframe
Terminology
Link Incident Record Registration (LIRR)
The LIRR Extended Link Service (ELS) requests that the recipient add the requesting port to its list of ports
that are to receive a Registered Link Incident Report (RLIR).
Registered Link Incident Report (RLIR)
RLIR ELS provides a way for a node port to send an incident record to another node port.
Request Node Identification Data (RNID)
RNID ELS acquires the associated nodes identification data, which provides configuration discovery and
management purpose information.
Missing Interrupt Handler Primary Timeout (MIHPTO)
A value and which determines how long the channel will wait before timing out an expected response from
the CUP.
What is FICON?
FIbre CONnection (FICON) is:
- An I/O protocol based on Fibre Channel (FC-SB-2 and FC-SB-3) used with IBM (and compatible)
mainframes and storage
- A layered protocol based on industry standards for Fibre Channel (FC) architecture using FC layer 4.
- The successor to ESCON
Introduced in the late 1990s, the original FICON adapter card which enabled FICON was developed for the
9672 G5 and G6 mainframes
FICON Frame Encoding
FICON frame encoding can be either FC standard 8b/10b used for 1, 2, 4 and 8 Gbps or 64b/66b with 10
Gbps ports. There are two types of frame encoding:
8b/10b encoding
64b/66b encoding
To use 64b/66b encoding, you must use a 10 Gbps ISL which cannot auto-negotiate with 1, 2, 4 and 8 Gbps
ports.
FICON Operating Modes
There are two FICON operating modes: FCV and FC. For System zSeries and 9672 G5 and G6 servers, there
were two supported modes:
FICON Bridge referred to as FCV which is a FICON channel mode designed to enable access to ESCON
interfaces using the FICON Bridge Adapter in the ESCON Director (9032-5)
FICON Native referred to as FC which enables access to the FICON channel mode from native FICON
control units in a point-to-point mode through a FICON Director (No cascaded FICON)
For zSeries and System z including z9 and z10, support was added for:
FICON Native (FC), but not FICON Bridge
Supported topologies: point-to-point, switched point-to-point, and cascaded FICON Directors
Fibre Channel Protocol (FCP) which provided support for open systems using industry standard SCSI
devices
FCP and FICON Mode Characteristics
Probably the single largest difference between the FICON channel (FC and FCV) and FCP channel mode types
is the treatment of data access control and security. FICON and ESCON channels rely on MIF to address
concerns regarding shared channels and devices. MIF provides ultra-high access control and security of data
so that one operating system image and its data requests cannot interfere with another operating systems
data requests. With the introduction of the System z, MIF continues this ultra-high level of access control and
security across channel subsystems (CSSs).
Linux guest operating systems under z/VM can have access to an FCP channel defined to the z/VM operating
system. Using MIF, an FCP channel can also be shared between Linux Logical Partitions and z/VM Logical
Partitions with Linux guests.
The FCP industry-standard architecture does not exploit the data access control and security functions of MIF.
As a result, FCP has the following limitations:
Channel sharing. When N_Port ID Virtualization (NPIV) is not implemented, and if multiple Linux images
share an FCP channel, and all of the Linux images have connectivity to all of the devices connected to the
FCP fabric, then all of the Linux images use the same worldwide port name to enter the fabric and, thus,
are indistinguishable from each other within the fabric. Hence, the use of zoning in switches and LUN
masking in controllers cannot be effective in creating appropriate access controls among the Linux
images.
By using NPIV, each operating system sharing an FCP channel is assigned a unique WWPN. The WWPN
can be used for device-level access control in storage controllers (LUN masking) and in switch-level
access control (zoning).
Device sharing. Without using NPIV, an FCP channel prevents logical units from being opened by more
than one Linux image at a time. Access is on a first come, first served basis. This is done to prevent
problems with concurrent access from Linux images that are sharing the same FCP channel and thus the
same worldwide port name. This usage serialization means that one Linux image can block other Linux
images from accessing the data on one or more logical units, unless the sharing images (z/VM guests)
are all well-behaved and not in contention.
FICON channels in FCP mode use the Queued Direct Input/Output (QDIO) architecture for communication with
the operating system. The QDIO architecture for FCP channels derives from the QDIO architecture that initially
was defined for the OSA-Express features and for HiperSockets communications. FCP channels do not use
control devices. Instead, data devices that represent QDIO queue pairs are defined, consisting of a request
queue and a response queue. Each queue pair represents a communication path between an operating
system and the FCP channel. It allows an operating system to send FCP requests to the FCP channel via the
request queue. The FCP channel uses the response queue to pass completion indications and unsolicited
status indications to the operating system. Figure2 on page5 illustrates this process.

FIGURE 2 I/O definition comparison (FCP to FICON and ESCON)
Channel Subsystem
The key to moving data into and out of a mainframe is the channel subsystem (CSS). When an I/O operation is
required, the CSS is passed the request from the main processor to manage. While awaiting completion of an
I/O request from the CSS, the main processor can continue processing other data. This process supports
concurrent transactions because of the interaction between the host processor and the CSS.
FIGURE 3 Channel Subsystem Overview
Multiple CSS Concept
The design of System z systems offers considerable processing power, memory size, and I/O connectivity. In
support of the larger I/O capability, the CSS concept has been scaled up correspondingly to provide relief for
the number of supported logical partitions, channels, and devices available to the system.
A single channel subsystem allows the definition of up to 256 channel paths. To overcome this limit, the
multiple channel subsystems concept was introduced. The architecture provides for up to four channel
subsystems. The structure of the multiple CSSs provides channel connectivity to the defined logical partitions
in a manner that is transparent to subsystems and application programs, enabling the definition of a
balanced configuration for the processor and I/O capabilities.
Each CSS can have from 1 to 256 channels and be configured to 1 to 15 logical partitions. Therefore, four
CSSs support a maximum of 60 logical partitions. CSSs are numbered from 0 to 3 and are sometimes
referred to as the CSS image ID (CSSID 0, 1, 2 or 3). These CSSs are also referred to as logical channel
subsystems (LCSSs).
Logical Channel Subsystem
The z/OS operating system is limited to a maximum of 256 channel paths (CHPIDs) for use within a logical
partition (LPAR). To facilitate the usage of more CHPIDs, the mainframe architecture supports a logical
channel subsystem (LCSS). The LCSS is functionally identical to the channel subsystem but up to four LCSSs
can be defined within a central processor complex. CHPIDs are unique within the LCSS only; consequently, the
256 CHPID limitation can be overcome. Usage can be maximized with NPIV and good fan-in/fan-out ratios.
Channel Spanning
Channel spanning extends the MIF concept of sharing channels across logical partitions to sharing physical
channels across logical partitions and channel subsystems.
Spanning is the ability for a physical channel (PCHID) to be mapped to CHPIDs defined in multiple channel
subsystems. When defined that way, the channels can be transparently shared by any or all of the configured
logical partitions, regardless of the channel subsystem to which the logical partition is configured.
A channel is considered a spanned channel if the same CHPID number in different CSSs is assigned to the
same PCHID in IOCP, or is defined as spanned in HCD.
In the case of internal channels (for example, IC links and HiperSockets), the same applies, but with no PCHID
association. They are defined with the same CHPID number in multiple CSSs.
Open Exchange
An open exchange is part of FICON (and FC) terminology. Many I/O operations can be in progress over FICON
channels at any one time. For example, a disk I/O operation might temporarily disconnect from the channel
while performing a seek operation or while waiting for a disk rotation. During this disconnect time, other I/O
operations can be managed, as explained here:
Command mode open exchanges
In command mode, the number of open exchanges is limited by the FICON Express feature. FICON
Express8, FICON Express4, FICON Express2 allow up to 64 open exchanges. One open exchange (actually,
this is an exchange pair) in command mode is the same as one I/O operation in progress.
Transport mode open exchanges
In transport mode, one exchange is sent from the channel to the control unit. Then the same exchange ID
is sent back from the control unit to the channel to complete the I/O operation. The maximum number of
simultaneous exchanges the channel can have open with the CU is up to 750 exchanges. The CU sets the
maximum number of exchanges in the status area of the transport mode response IU (the default number
is 64 and can be increased or decreased).
FICON Connection Components
FIGURE 4 FICON Connection Components
FICON Express Cards
Original FICON card:
- 100 MB/sec full duplex
- LX: supports 9 micron SM fiber
- SX supports 50, 62.5 MM fiber
- FICON Express2 (LX):
- Supports 9 micron SM fiber
FICON Express2 (SX):
- Supports 50, 62.5 micron MM fiber
- Supported on all z9 processors
FICON Express4 (LX and SX):
- 400 MB/sec full duplex (the IBM certified maximum data rate capability for FICON is 350 MB/sec)
- Supports 9, 50, 62.5 micron SM and MM fiber
- Supported on System z9 and z10 servers
- FICON Express4-2C adapters are supported on System z9 BC servers only, and go 550m or 4km
FICON Express2, FICON Express4, FICON Express8, and FICON Express8S features do not support FCV
mode. FCV mode is available with FICON Express LX feature 2319 (carry forward only on System z10). FCV
was used with the FICON Bridge solution.
FICON Express8 (LX and SX):
- 800 MB/sec full duplex.
- Supports 9 micron SM fiber and 50, 62.5 micron MM fiber.
- Supported only on System z10 servers.
- Support for FC and FCP modes.
- Direct memory access using the QDIO architecture.
FICON Express8S is the most current FICON card
- Allows for the consolidation of existing FICON Express, FICON Express2, FICON Express4, and FICON
Express8 channels onto fewer FICON Express8S channels while maintaining and enhancing
performance.
- Offers performance improvements for High Performance FICON for System z (zHPF), and Fibre
Channel Protocol (FCP) to ensure the zEnterprise EC12, zEnterprise 196 and zEnterprise 114 servers
continue to allow your bandwidth to increase to meet the demands of your business applications.
- Operates at 2, 4, or 8 Gbps autonegotiated.
- Has a feature supporting single mode fiber optic cabling (9 micron) terminated with an LC Duplex
connector - FICON Express8S 10KM LX.
- Has a feature supporting multimode fiber optic cabling (50 or 62.5 micron) terminated with an LC
Duplex connector - FICON Express8S SX.
FICON Express8S channels
- Each zEC12, z196, or z114 FICON Express8S channel has two channels (ports) per feature.
- Each FICON Express8S channel has two possible modes of operation designed for connectivity to
servers, switches/directors, disk, tape, and printers:
CHPID type FC FICON, zHPF, and channel-to-channel (CTC) traffic for the z/OS, z/VM, z/VSE, z/
TPF, and Linux on System z environments
CHPID type FCP - Fibre Channel Protocol (FCP) for attachment to SCSI devices for the z/VM, z/VSE,
and Linux on System z environments
The maximum distance for a single director is 10km from the host mainframe. If you are trying to determine
the maximum distance of a device attached to the director to the mainframe, you need to add in that distance
as well. For example, a director could be attached to a mainframe that is 5km away and a storage device is
attached to the Director which is also 5km away. The total distance then that the storage device can be from
the mainframe is 10km.
Brocade MLX Series Multiservice Ethernet routers
The Brocade MLX routers provide a rich set of high performance IP version 4, IP version 6, and Multiprotocol
Label Switching (MPLS) and advanced Layer 2 switching capabilities. As a result, these routers address the
diverse needs in environments that include service provider backbones, data centers, and distributed
enterprises.
The Brocade MLX routers are highly optimized for IP Ethernet deployments, providing symmetric scaling with
chassis options that include 4-, 8-, 16-, and 32-slot systems. These routers offer industry-leading wire-speed
port capacity without compromising the performance of advanced forwarding capabilities. For example, the
Brocade MLXe-32 delivers data forwarding performance in excess of 6 Tbps today and scales to 15.36 Tbps.
This capacity is enough to future proof networks for years to come. In addition, the robust control plane is
proven in thousands of mission-critical deployments around the globe.
The Brocade MLX routers enable reliable converged infrastructures and support key applications. These
routers feature advanced redundant switch fabric architecture for high availability. The architecture helps
ensure that the system continues to operate at peak performance even if a switch fabric card failure happens.
If additional fabric failures occur (although unlikely), with the advanced architecture, the system continues
operating in a graceful degradation mode. In this mode, the system tunes its performance to the remaining
fabric capacity.
The Ethernet line modules for the Brocade MLX routers are available in two different types:
The 1-GbE line modules are available in 24- and 48-port models
10 GbE line modules are available in 4-port and 8-port models
The Brocade MLX routers use Brocade Network Advisor application, which offers comprehensive unified
network management for all Brocade products. Brocade Network Advisor provides an easy-to-use MPLS
Manager. MPLS Manager can help to configure, monitor, and manage services across networks. Brocade
Network Advisor also uses sFlow-based technology that provides the following capabilities:
Proactive monitoring
Traffic analysis
Reporting
Brocade Network Advisor helps to reduce network downtime. In addition, Brocade Network Advisor offers
administrators end-to-end network visibility from a single dashboard.
z/OS Discovery and Auto-configuration
z/OS discovery and auto-configuration (zDAC) is a function, exclusive to zEC12, z196 and z114, that
automatically performs a number of I/O configuration definition tasks for new and changed disk and tape
controllers that are connected to a FICON Director. It is designed to help simplify I/O configuration of
zEnterprise CPCs running z/OS, and to help reduce complexity and setup time.
The zDAC function is integrated into the existing Hardware Configuration Definition (HCD) tool. A policy can be
defined in HCD according to the availability and bandwidth requirements, including parallel access volume
(PAV) definitions, control unit numbers, and device number ranges. The zDAC proposed configurations are
created as work I/O definition files (IODF) that can be converted to production IODF and activated.
zDAC provides real time discovery for the FICON fabric, subsystem, and I/O device resource changes from z/
OS. By exploring the discovered control units for defined LCUs and devices, zDAC compares the discovered
controller information with the current system configuration to determine delta changes to the configuration
for a proposed configuration. All new added or changed logical control units and devices will be added into the
proposed configuration with proposed control unit and device numbers and channel paths based on the
defined policy. zDAC uses a channel path chosen algorithm to minimize single point of failure. zDAC applies to
all FICON features supported on zEC12, z196 and z114 when configured as CHPID type FC.
Parallel Sysplex and Coupling Links
The Parallel Sysplex technology represents a synergy between hardware and software and comprises Parallel
Sysplex-capable servers, Coupling Facility, coupling links (IFB, IC, ICB, and ISC-3), Server Time Protocol (STP),
optionally Sysplex Timers2, shared DASD, and software, both system and subsystem, designed for parallel
processing as shown in Figure5.
FIGURE 5 Parallel Sysplex connectivity with mixed Coordinated Timing Network (CTN)
Parallel Sysplex technology is a highly advanced, clustered, commercial processing system. It supports high-
performance, multisystem, read/write data sharing, enabling the aggregate capacity of multiple z/OS systems
to be applied against common workloads.
The systems in a Parallel Sysplex configuration are linked and may fully share the same devices and run the
same applications. This allows users to harness the power of multiple zEC12 and System z systems as if they
were a single logical computing system.
The architecture is centered around the implementation of a Coupling Facility running the Coupling Facility
Control Code (CFCC) and high-speed coupling connections for intersystem and intrasystem communications.
The Coupling Facility is responsible for providing high-speed data sharing with data integrity across multiple
zEC12 and System z servers.
Parallel Sysplex is designed to provide high availability for business critical applications. The design is further
enhanced with the introduction of System-Managed Coupling Facility Structure Duplexing which provides the
following additional benefits:
Availability: Structures do not need to be rebuilt in the event of a coupling facility failure
Manageability and usability: A consistent procedure is established to manage structure recovery across
exploiters
Configuration benefits: A sysplex can be configured with internal CFs only
A parallel Sysplex supports connectivity between systems differing by up to two generations. For instance, an
IBM System z9 cannot participate in a Sysplex with a zEC12, but zEnterprise systems and z10 systems can.
zHPF - High Performance FICON
With the introduction of High performance FICON (zHPF), the FICON architecture has been streamlined by
removing significant overhead to the storage subsystem and the microprocessor within the FICON channel. A
command block is created to chain commands into significantly fewer IUs. The overhead required to convert
individual commands into FICON format is removed as multiple System z I/O commands are packaged
together and passed directly over the fibre optic link. For zHPF operations, we lose the concept of the
Command Control Word (CCW) and it is replaced with the Transport Control Word (TCW). The TCW allows
multiple channel commands to be sent to the control unit as a single entity instead of a stream of individual
commands. Compatibility is established such that existing FICON operations (CCWs) coexist with zHPF
operations (TCWs).
Essentially, zHPF reduces the number of information units (IUs) processed. Enhancements have been made
to the z/Architecture and the FICON interface architecture to provide optimizations for OnLine Transaction
Processing (OLTP) workloads.
On the zEC12 and z196 the multitrack operations extension applies exclusively to the FICON Express8S,
FICON Express8, and FICON Express4
1
, when configured as CHPID type FC, and connecting to z/OS. zHPF
requires matching support by the DS8000 series, otherwise the extended multitrack support is transparent to
the control unit.
From the z/OS point of view, the existing FICON architecture is called command mode and zHPF architecture
is called transport mode. During link initialization, the channel node and the control unit node indicate
whether they support zHPF.
ATTENTION
All FICON channel paths (CHPIDs) defined to the same Logical Control Unit (LCU) must support zHPF. The
inclusion of any non-compliant zHPF features in the path group will cause the entire path group to support
command mode only.
1. FICON Express4 LX 4KM is not support on zEC12.
The mode used for an I/O operation depends on the control unit supporting zHPF and settings in the z/OS
operating system. For z/OS exploitation there is a parameter in the IECIOSxx member of SYS1.PARMLIB
(ZHPF=YES or NO) and in the SETIOS system command to control whether zHPF is enabled or disabled. The
default is ZHPF=NO.
Support is also added for the D IOS,ZHPF system command to indicate whether zHPF is enabled, disabled, or
not supported on the server.
Brocade Network Advisor
Most IT organizations that use IBM zEnterprise and TS7700 Grid networks face similar challenges in
managing their networks. These challenges include minimizing downtime and managing application service-
level agreements (SLAs). At the same time, they must find ways to reduce operational expenses. As networks
become increasingly complex and critical to business operations, many IT organizations rely on multiple point
solutions that address only parts of the network. This network complexity leads to cumbersome, error prone
operations. For example, you might have separate management tools (and servers) for each of the
components that are illustrated in Figure6.

FIGURE 6 IBM and Brocade Integrated TS7700 Grid Network
Brocade Network Advisor is the industrys first unified network management solution for data, storage, and
converged networks. Brocade Network Advisor supports various networks, including the following examples:
Fibre Channel SANs including 16 Gbps platforms
Fibre Channel over Ethernet networks
Layer 2/3 IP networks, application delivery networks
MPLS networks in service provider environments
Brocade Network Advisor is an ideal tool to use for managing a TS7700 Grid network. Brocade Network
Advisor gives the user the ability to manage all of the TS7700 Grid network hardware components from a
single user interface by using a single server. With Brocade Network Advisor, organizations have a single
management framework for their entire network infrastructure. Brocade Network Advisor has intelligently
designed interfaces and full role-based access control (RBAC) to support the needs of different network
teams.
NOTE
FMS must be enabled on the local switch for the remote CUP to work.
Brocade Network Advisor can be used to manage storage and data networks separately or through a single
instance of the product. This capability enables better coordination between storage and data networking
administrators that perform provisioning, troubleshooting, and reporting. Brocade Network Advisor also helps
organizations reduce network downtime through the following capabilities, among other types:
Real-time and historical performance monitoring
Traffic analysis
Change management
Policy driven remedial actions
Proactive alerts with real-time logging, diagnostic, and fault isolation capabilities help resolve issues before
they affect SLAs. For more granular performance analysis, Brocade Network Advisor uses the hardware-based
sFlow technology. The sFlow technology is available on all Brocade IP switches and routers. The sFlow
technology provides real-time network monitoring and accounting capabilities without affecting network
performance. The Traffic Analyzer feature provides trend analysis, management, and monitoring tools for
sFlow reporting, accounting, and presentation for all IP devices. Brocade Network Advisor also offers open,
standards-based interfaces. It also provides deep integration with a wide range of third-party network
monitoring, virtualization, wireless, and security solutions. It integrates with leading server and storage
automation solutions to bridge operational gaps across server, network, and storage administrators. This
integration provides end-to-end network visibility through frameworks such as the Storage Management
Initiative-Specification (SMI-S). Of substantial benefit to the TS7700 Grid user, Brocade Network Advisor
provides support for leading data center orchestration frameworks, including IBM Tivoli Storage Productivity
Center (TPC). Brocade Network Advisor and IBM TPC integration is focused on administrators within the same
domain, in this case, storage and SAN management. The IBM TPC solution is an industry-leading storage
resource management solution that helps customers with the management of storage environments. It
simplifies various tasks that are related to the following capabilities:
Logical unit number provisioning
Capacity planning
Replication
Reporting
IBM TPC also performs basic SAN management activities for Brocade Fibre Channel infrastructure by using
the Brocade standards-based SMI-S interface.
Event Notification
The Management application records the SAN and IP events in the Master Log. You can configure the
application to send event notifications to e-mail addresses at certain time intervals. This is a convenient way
to keep track of events that occur on the SAN and IP networks. You can also configure products to call home
for certain events, notifying the service center of product problems.
Filtering Event Notifications
The application provides notification of many different types of SAN events. If a user wants to receive
notification of certain events, you can filter the events specifically for that user.
Reasons to Migrate from ESCON
FICON is an improved protocol over the ESCON protocol
- ESCON provides a single I/O per channel whereas FICON uses multiple, simultaneous ESCON-type I/
Os
- ESCON I/O is half duplex; FICON I/O is full duplex
- FICON provides improved support over distance decreased data droop
- FICON offer channel path consolidation
- FICON provides increased addressing
- FICON offers a bandwidth increase
- Buffer credit control
- FICON adapters can support Open Systems and LINUX (FICON/FCP intermix)
- zHPF support
FICON uses Open Systems components
- Fibre Channel Directors
- Fibre Channel cables & connectors
- Fibre Channel adapters
Migration Considerations:
- Buffer credits
- SX/LX requirements as applicable
FICON CUP Support
Control Unit Port (CUP) is an embedded and not a physical port. It allows in-band management of directors
from the management applications on a mainframe and is u.sed for configuration, monitoring, and error
handling.
CUP support is provided by all Brocade certified FICON switches and Directors
- Optional licensed feature for both B-Series and M-Series
- FICON Management Server license required to activate CUP
- Supported in single or cascaded FICON environments
- Utilizes embedded port FE
CUP is typically used by customers with:
- SA z/OS (System Automation software) configuration tool
- RMF (Resource Measurement Facility) monitoring tool
If a mainframe administrator wants to be able to do in-band management from their Hardware
Management Consoles (HMC), they need CUP
If it is important to them to get the Service Information Messages (SIM) to the z/OS console for FICON
device hardware failures
- Interpret these messages using the IBM Mainframe Systems Messages and Codes manual
- If there is a need to do RMF reports for reporting on FICON Director activity statistics in the Record
Type 74 Subtype 7, CUP is mandatory
- RMF option FCD/NOFCD in ERBRMFxx enables or disables FICON CUP Director
Although IBM has developed a new machine type, 2027, for FICON Directors, all Directors and switches,
regardless of model or manufacture should be configured as 2032
Active=Saved will prevent a loss of connectivity on a Director using CUP
FICON Directors have the embedded port FE for the CUP port. On FICON Directors that have 256 or more
ports, this has caused a slight dilemma. This logical FE overlaps the physical FE port, so the physical FE
and FF ports cannot be used on these Directors for FICON connectivity. They may still be used for port swaps
or for FICON/FCP intermix, however, they cannot be genned in HCD for FICON connectivity.
In a FICON environment, only one RMF LPAR should attempt to access the CUP port at any one time, and it is
still best practice to have two or more CHPIDs with access to the CUP. However, too much activity to the FICON
CUP can cause missing interrupts, leading to the potential for having a boxed device.
FIGURE 7 CUP
2
2. Footnote 1 from Figure7 on page17: Safe switching is the ability to manipulate ports and adjust path
status non-disruptively. In order for System Automation I/O operations to provide safe switching, the System
Automation I/O must have access to all switches. All the switches must be online as I/O devices on all the
systems where System Automation is running in order to verify that a path can be removed before the path is
actually removed.
This is a simplified representation of an FICON environment and how CUP fits into that environment. Best
practice is to have two or more CHPIDs with access to the CUP and implement safe switching. Safe switching
describes the System Automation process of path removal. Safe switching is a process that involves
contacting all systems in a sysplex and verifying a path can be removed before it is actually removed.
Features of CUP
Remote switch functions via z/OS System Automation
Reporting of failed FRUs to z/OS
Needed to add Directors to the I/O sysgen for error reporting purposes
May require the CUP device be taken offline before a firmware upgrade
Prohibits communication between port pairs
Allows the host to set the switch to an offline state
Prohibit Dynamic Connectivity Mask (PDCM)
CUP also provides PDCM as a management tool. The PDCM is a mechanism to define port connectivity
such as prohibits, allows, and blocking/unblocking of ports.
The FICON Prohibit Dynamic Connectivity Mask (PDCM) controls whether or not communication between a
pair of ports in the switch is prohibited or allowed
Block versus Prohibit:
- Blocking causes the firmware to send a continuous offline sequence to the port
Useful to report the link as inactive after varying a device off on the mainframe
- Prohibit causes the firmware to prevent connectivity between the ports
Useful to force FICON traffic over specific ISLs
PDCM controls whether or not communication between a pair of ports in the switch is prohibited or
allowed
If there are any differences in restrictions set up with Brocade Advanced Zoning and PDCM, the most
restrictive rules are automatically applied
Mainframe Reporting with RMF
As mainframes have grown, monitoring resources to provide peak performance has become increasingly
complex. Resource Measurement Facility (RMF) provide tools for performance measurement and
management for a single z/OS system or a system complex of z/OS systems:
RMF is used to gather data, report data, and access data across a sysplex from a central administration
point.
RMF is used to track problems back to their sources, no matter where the problem occurred within the
enterprise to enable administrators to respond before the problems become critical.
RMF when used with CUP, can gather data about FICON Directors and make that data available to the
mainframe system administrator thus providing a single point of administration.
CUP, or Control Unit Port is a holdover from ESCON Directors. In a FICON environment, CUP allows for in-band
management and device statistics reporting. This opens the door to FICON director performance metrics
reported out via the RMF 74-7 record, more commonly known as the FICON Director Activity Report.
When you install FICON Management Server (FMS) on a FICON switching device, and then enable CUP on that
device, you provide yourself with a lot of valuable tools:
RMF reports on FICON fabric statistics such as Frame Pacing Delay (the only way to get this information
into RMF) and SA z/OS IO/OPs can have in-band access to the FICON switching devices and can manage
and monitor them. This is how to get switch related hardware errors to be reported to z/OS against a
device (domain ID) number. If the switch is not defined as an I/O device through the use of CUP, and/or
that I/O device is not online to z/OS, then switch related errors cannot be surfaced up to z/OS so that it
can take appropriate actions.
Systems Automation can monitor and manage fabric components
Date and Time can be set by z/OS for log and event synchronization
The Allow/Prohibit Addressing Matrix becomes available
The Active=Saved condition becomes user configurable
At FOS 7.0+, the Port De-Commissioning Feature will be able to function
Creating RMF 74-7 Records
Enabling RMF 74 subtype 7 (RMF 74-7) records, used for switches and directors, yields an RMF report called
the FICON Director Activity Report. RMF 74-7 records are collected for each RMF interval if:
The FMS license is installed on the switching device
The FMS indicator has been enabled on the switch
IOCP has been updated to include a CNTLUNIT macro for the FICON switch
IECIOSnn parmlib member is updated to include FICON STATS=YES
FCD is specified in the ERBRMFnn parmlib member
- FCD/NOFCD can also be modified through an operator command
Only a single LPAR in the sysplex should be gathering these RMF statistics:
The data contained in a FICON switch is systems level data
The master RMF LPAR will distribute this data to other LPARs
Relieves congestion due to having all LPARs asking directly for this data
To use CUP with Systems Automation, the IO Operations (IOOPs) module must be licensed and implemented.
FICON CUP and FOS Firmware Upgrades
During the FICON Director firmware download process a failover to the backup processor occurs. Any
outstanding CUP commands are lost and result in a MIHPTO timeout (usually 3 minutes). Inefficient usage of
MIHPTO can result in device boxing. If the CUP falls behind in honoring requests it begins getting strikes
against it at the host programming level and after three strikes, the CUPs switch will be boxed. This occurs
when upgrading firmware on a FICON switching device. To resolve the boxing issue, vary the CUP port 0xFE
offline before starting the firmware upgrade and when complete vary the CUP port back online.
The ficoncupset mihpto command can be used to increase, decrease, or display the MIHPTO value that
the CUP returns to the host when configuration data is read. Changing the MIHPTO value requires that the
CUP control device be varied offline from the host, as the host will not automatically be notified when the
value is changed. The host will normally read the new value, using the Read Configuration Data command,
when the CUP is varied back online.
2 Design Considerations
After completing this section you should be able to describe and demonstrate your knowledge of the following:
Single vs. cascaded director design requirements
Scalability considerations
Knowledge of Virtual Fabrics
Factors that influence FICON performance
High Availability (HA) and disaster recovery solutions
FICON and FCP intermix configurations
Extension solutions
Single Director vs. Cascaded Director Design Requirements
FIGURE 8 Single vs. Cascaded FICON Director Environments
A hop is defined as one ISL connection. More than one ISL connection is not supported between directors.
Therefore, you can have two directors connected together through an ISL but not more than two directors. In
this example, the cascaded configuration shows how an open system server can also be included in the
configuration as the open system server is attached to one director that is attached to another director.
In open systems, connecting two Directors together is achieved by creating an Inter-Switch Links (ISLs)
between the machines. In the FICON environment, this concept of connecting two Directors together is
called cascading. M-Series Directors use preferred path.
The standard FC routing mechanism for cascaded links is called Fabric Shortest Path First (FSPF).
The FSPF protocol compares the cost of various paths between a source Director and a destination
Director by adding the costs of the ISLs along each path. FSPF chooses the path with minimum cost. If
multiple paths exist with the same minimum cost, FSPF distributes the load among these paths.
- Database maintained of ISL and ICL link states
- Reliable flooding is not the method of synchronization between two switches
Required Fabric OS settings are dlsreset and iodset
- iodset causes a delay to make sure frames have left the fabric
On M-Series Directors. the rerouting delay can lead to I/O timeouts
TABLE 1 Brocade enterprise-class platform blade terminology
ATTENTION
The 10 GbE SFPs used in the FX8-24 blade and the 10 Gbps FC SFPs used in the FC16-32/48 blades are NOT
interchangeable.
Term Abbreviation Blade ID Definition
32-port 16-Gbps port blade FC16-32 97 A 32-port Brocade platform port blade supporting 2,
4, 8, 10, and 16 Gbps port speeds.
With Virtual Fabrics (VF) enabled, you can create
up to eight 32-port domains.
NOTE: 10 Gbps speed for FC16-xx blades requires
the 10G license.
DCX Extension blade FX8-24 75 A 24-port Fibre Channel routing and FCIP blade that
also has 10 1-GbE and two 10-GbE ports and is
compatible with the Brocade DCX, DCX-4S, and
DCX 8510 family CP blades.
With VF enabled, you can create up to eight 48-
port domains
Up to 4 FX8-24 blades are supported in the DCX
8510 chassis using FOS v7.0.
Connectivity Blades
48-port blades require Interop Mode 0 (IM=0), so you cannot use a mix of B-Series and M-Series in a
fabric:
- Eight slots filled with 48-port blades would put 384 ports into a single domain ID
- IBM z/OS only allows up to 256 ports to be within a single domain ID
- This creates a requirement for the use of zero-based addressing and Brocade Virtual Fabrics on a
Brocade DCX
- Zero-based addressing is required, but it is not supported in IM=2 mode, only in IM=0 mode
- Therefore, on a Brocade DCX, zero-based addressing requires the use of Brocade Virtual Fabrics (VF)
thereby making VF a requirement when using 48-port blades on a Brocade DCX
- On a Brocade DCX-4S, the chassis has only four slots; filled with 48-port blades, it would put 192
ports into a single Domain ID
- Since z/OS allows up to 256 ports to be within a single Domain ID, the Brocade DCX-4S does not have
to be run using Brocade Virtual Fabrics (although VF can still be deployed if desired)
- On the Brocade DCX-4S, IM=2 was never qualified and, therefore, 48-port blade use is limited to IM=0
only
2-byte Link Addressing
ISLs are not used nor defined
The first byte is the Domain ID (CHPID) and the second byte is the switch port (egress port)
All CHPIDs with 2-byte link addresses require all links accessing the CHPID to be a 2-byte link
2-byte addressing enables cascaded switch support; the second byte enables a Domain ID to be specified
in the IOCP definition
Remember that intermixed IOCP definitions (1-byte and 2-byte link addresses) on the same channel path
are not allowed
When you create 2-byte FICON addresses in a mainframe configuration:
- The mainframe channels will issue the Query Security Attributes CCW at initialization time (after the
FLOGI) to see if the entry FICON Director supports what IBM commonly calls a high integrity fabric (2-
byte addressing not 1-byte addressing)
- The channel expects the QSA Response to contain two bits indicating fabric binding and insistent
Domain IDs (IDID) are configured and established throughout the entire fabric
- If these bits come back false (either one), then the channel stops and terminates the login process
- The QSA response will be affirmative:
IF and ONLY IF security is implemented on each Director in this fabric
Fabric Binding is enabled
Insistent Domain_ID is enabled
- If any of this is NOT true, then the QSA command will indicate a "high integrity fabric" is not available,
and the channel will not initialize
The only way to recover the channel at this point is to either use single byte addresses or turn on
Fabric Binding and Insistent domain ID for every switch in the fabric
The QSA command will not be sent if 1-byte addressing is used for a FICON Director and is in the CNTLUNIT
macro in the IOCDS. QSA is only used for cascaded-FICON or high integrity FICON. Anytime you see a 1-byte
address for a Director, you know that Director is not part of a cascaded environment.
FIGURE 9 2-Byte Link Addressing
Configuring CHPIDs Online
Two settings must be implemented to configure the CHPIDs online, fabric binding and insistent domain IDs. After
you successfully activate the new I/O definitions, you can configure the channel path online and try to communicate
to the devices.
Scalability Considerations
Trunking
The trunking feature optimizes the use of bandwidth by allowing a group of links to merge into a single logical
link, called a trunk group. Traffic is distributed dynamically and in order over this trunk group, achieving
greater performance with fewer links. Within the trunk group, multiple physical ports appear as a single port,
thus simplifying management. Trunking also improves system reliability by maintaining in-order delivery of
data and avoiding I/O retries if one link within the trunk group fails.
Trunking is frame-based instead of exchange-based. Since a frame is much smaller than an exchange, this
means that frame-based trunks are more granular and better balanced than exchange-based trunks and
provide maximum utilization of links.
Trunking combines two or more physical ISLs into a single logical link, while limiting FSPF rerouting
Trunk group characteristics:
- Frames are multiplexed across ISLs in the trunk group
- One port in the trunk group represents the link in the routing database
Trunking license required for all switches participating in trunking
Available when the license is installed and ports are reinitialize
Trunk ports must operate at a common port speed
Trunk ports must originate and end in valid port groups (i.e. 0-7, 8-15, etc.)
When trunking criteria is met the trunk forms automatically
Long distance trunks require Extended Fabric licenses on both ends of the link
FIGURE 10 Sample B-Series Trunk Group
High Availability
Buffer-to-Buffer Credits
Define the maximum amount of data that can be sent prior to an acknowledgement
Buffer credits are physical ASIC port or card memory resources and are finite in number as a function of
cost
Within a fabric, each port may have a different number of buffer credits
The number of available buffer credits is communicated at fabric logon (FLOGI)
One buffer credit allows a device to send a fixed 2112 byte frame of data (2K usable for z/OS data)
Assuming that each credit is completely full, you need one credit for every 1km of link length over a 2
Gbps fiber
Unfortunately, z/OS disk workload rarely produces full credits. For a 4K transfer, the average frame size
for a 4K transfer is 819 bytes.
Five credits would be required per km over a 4 Gbps fiber
The optimal amount of buffer credits is determined by the distance (frame delivery time), the processing
time at the receiving port, link signaling rate, and the size of the frames being transmitted:
Credit = (round-trip-time + receiving-port-processing time) / frame-transmission-time
Information required for calculation:
- Link speed (1, 2, 4, 10 Gbps)
- Actual fiber distance in km that the frame must traverse
- The size of the frame can be difficult to obtain
Formula for assigning buffer credits using 2148 frame size
- 1 Gbps where distance (in km) / 2 + 20%
- 2 Gbps where distance (in km) + 20%
- 4 Gbps where distance (in km) x 2 + 20%
- 10 Gbps where distance (in km) x 6 + 20%
If you always assume a frame size of 2148 bytes, in the vast majority of cases, it will produce too few
buffer credits
For extended links, RTT and the average frame payload size must also be considered
Long Distance Link Modes
Use the portcfglongdistance command to support long distance links and to allocate sufficient
numbers of full size frame buffers on a particular port. Changes made by this command are persistent across
switch reboots and power cycles. This command supports the following long-distance link modes:
Static Mode (LO) - L0 is the normal (default) mode for a port. It configures the port as a regular port. A
total of 20 full-size frame buffers are reserved for data traffic, regardless of the port operating speed;
therefore, the maximum supported link distance is up to 5 km at 2 Gbps, up to 2 km at 4 Gbps, and up to
1 km at 8, 10, and 16 Gbps.
Static Mode (LE) - LE configures an E_Ports distance greater than 5 km and up to 10 km. LE does not
require an Extended Fabrics license. The baseline for the buffer credit calculation is one buffer credit per
km at 2 Gbps. This yields the following values for 10 km:
- 10 buffer credits per port at 2 Gbps.
- 50 buffer credits per port at 10 Gbps
- 80 buffer credits per port at 16 Gbps
Dynamic Mode (LD) - LD calculates BB credits based on the distance measured during port initialization.
Brocade switches use a proprietary algorithm to estimate distance across an ISL. The estimated distance
is used to determine the BB credits required in LD (Dynamic) extended link mode based on a maximum
Fibre Channel payload size of 2,112. You can place an upper limit on the calculation by providing a
desired_distance value. Fabric OS confines user entries to no larger than what it has estimated the
distance to be. When the measured distance is more than desired_distance, the desired_distance (the
smaller value) is used in the calculation.
Static Long-Distance Mode (LS) - LS calculates a static number of BB credits based only on a user-defined
desired_distance value. LS mode also assumes that all FC payloads are 2112 bytes. Specify LS mode to
configure a static long distance link with a fixed buffer allocation greater than 10 km. Up to a total of 1452
full-size frame buffers are reserved for data traffic, depending on the specified desired_distance value.
FICON Emulation Requirement for a Determinate Path
FICON emulation processing creates FICON commands and responses on extended CHPIDs, and intercepts all
exchanges between a channel and a CU. For FICON Emulation processing to function correctly, all the
exchanges between a channel and CU must take a single tunnel path.
There are two ways to ensure a determinate path for FICON commands and responses:
Define only one FCIP tunnel between sites
Use a Traffic Isolation zone (TI Zone) to assign a specific tunnel to Channel and control unit ports. In the
cases where multiple tunnels are required between a pair of FICON switches (or Logical Switches), use TIZ
to define ports that should use specific tunnels.
FCIP Tunnel Between Sites
The Brocade 7800 and FX8-24 uses FCIP Trunking features to overcome the limitation of one Ethernet
interface, one IP address and one FCIP tunnel. In Fabric OS v6.3 and later an FCIP Tunnel is created with
multiple FCIP circuits over different IP interfaces to provide WAN load balancing and failover recovery in the
event of a limited WAN outage. This provides a highly redundant WAN configuration for all FICON or FCP
emulation technologies with Fabric OS. Figure11 shows that a tunnel can have up to 4 circuits per GbE port.
FIGURE 11 FCIP tunnel and circuits
Brocade FCIP Considerations
APTpolicy must be supported for FICON if more than one VE_Port must be used. Multiple FX blades will
require more than one VE_Port. Only certain APT policies are supported for FICON, such as PBR (Port Based
Routing), however, with FCIP, multiple VE_Ports between the same two domains is not supported because
when there are routing changes it can cause an IFCC.
Data in-flight will be lost (IFCC) under these circumstances:
Modifying or deleting a circuit or tunnel
The Lossless feature with PBR does not extended beyond the C2/C3 ASICs internal to the 7800/FX blade,
therefore, routing changes that occur will result in data loss across the FCIP ISLs thus causing an IFCC. If
multiple FCIP complexes are required, such as obtaining 20 Gbps on a single blade or using multiple FX
blades in a single 8510 chassis, this is not supported because FCIP Trunking cannot be configured across
FCIP complex and across FX blades. A recommended alternative architecture would be to utilize multiple
VF LS with a single VE_Port in each. Two LSs with one VE_Port in each can fully utilize both FCIP
complexes on an FX blade. Channels will have to be distributed across the two LSs.
FOS FW upgrade/downgrade
Firmware upgrades/downgrades and modifying or deleting a circuit or tunnel in which circuits belong to,
can result in an IFCC because data being transmitted will be lost during those operations.
Data will be delivered out of order (IFCC) under these circumstances:
Protocol optimization with more than one VE_Port between the same domains
Protocol optimization is only supported when there is a single VE_Port connecting two domains. This
establishes a deterministic path for data that must pass through the same VE_Ports outbound as it does
on the return. The path is not deterministic if two fabrics are connected with multiple links as the fabric
may choose to send data out on one path and return on a different path, which must have different
VE_Ports. If the VE_Ports are not consistent the result will be breaking the protocol optimization state
machine. FCIP Trunking solves this problem by allowing multiple connections while using a single VE_Port
and always delivering data in-order.
Wrong APTpolicy with more than one VE_Port between domains
The following are considerations when connecting Virtual Fabric logical switches between sites:
The FCIP tunnel should be located in its own logical switch
The XGE interface must be in the default switch
There must be a VE_Port in each environment
VLAN tagging is done per FCIP circuit
High Performance FCIP
PP-TCP-QoS is a special QoS function exclusive to Brocade extension and developed especially for high-
performance FCIP. For QoS to function properly across an IP network, it is essential that each priority have its
own flow. This is not possible using only a single TCP session, because there would only be a single merged
flow. PP-TCP-QoS provides a TCP session for each priority flow, so that the IP network can perform according to
the QoS settings. FCIP Trunking is fully compatible with PP-TCP-QoS.
7800/FX support three levels of priority (H/M/L). The default amount of bandwidth (BW) that the scheduler
will apportion during times of contention is 50/30/20%. QoS portioning of BW only occurs during times of
contention, otherwise, the BW is shared equally across all priorities. It is possible to change the default
portions to any values you wish as long as H>M>L and the aggregate of all the priorities equals 100%.
There are four TCP sessions per FCIP circuit H, M, L, and F-class. F-Class uses a strict queuing which means if
there is any F-Class to send, it all gets sent first. There is very little F-Class traffic and it does not interfere with
data traffic. Each TCP session is autonomous and does not rely on other TCP sessions or settings. Each TCP
session can be configured with its own DSCP, VLAN tagging and 802.1P values. This permits that TCP session
(priority) to be treated independently in the IP network from site-to-site based on the SLA for that QoS priority.
Traffic Isolation Zoning
The Traffic Isolation Routing feature may be used to control the flow of interswitch traffic through the Brocade
extension switch or blade VE_Port or E_Port connections. Traffic Isolation Routing uses a special zone, called
a Traffic Isolation zone (TI zone), to create dedicated paths for specific traffic.
When setting up TI Zones in your FICON environment, it is not recommended to have failover enabled for the
following reasons:
FICON Emulation will not occur on a failed over path
FICON devices are identified at one time during channel path activation (ELP/LPE exchange) if the
paths do not fail, the MVS system will not re-issue this sequence over the recovery path
Multiple error periods will be perceived at the connected LPARs (first link failure, movement of traffic to
recovery path and then again when the primary path is restored)
There is no good way to perform a forced or controlled fallback
Keep the following best practices in mind when using TI zoning:
If TI Zones are used to provide routes for Emulated traffic then TI Zone Fail-over cannot be enabled
Keep the configurations as simple as possible, this will make the systems more supportable
Include virtual E_Ports in the Traffic Isolation zone
Use TI zoning for route selection on a port-to-port basis.
Use port zoning to restrict data flows
Look at TI zoning, link costs, and zoning definitions to understand the routing paths used in your data
center.
Understand bandwidth requirements, available network resources, etc
Collect information on all new installations or after network changes by running the
portCmd --ipPerf command
Separate disaster recovery networks from production networks
Enhanced TI Zoning
Enhanced Traffic Isolation Zones allow you to specify a preferred and exclusive path through a cascaded
fabric for a particular flow by defining a set of devices or ports to appear in more than one Enhanced TI Zone.
A preferred path is one that has failover enabled and an exclusive path has failover disabled. An exclusive
path in the following topologies is beneficial as it will limit the number of hops between directors:
Triangular
System Data Mover
Emulation
Enhanced TI Zoning for the topologies listed above is available only on the following platforms running Fabric
OS v6.4.0 or later:
Brocade 5100
Brocade 5300
Brocade 7800
Brocade DCX
Brocade DCX-4S
Brocade 8510
FC Long Distance Cascading
Outside of a local data center, FICON cascading is for connectivity. Connecting FICON attachments between
data centers allows for the traffic flow of data between those sites especially for business continuance and/or
disaster recovery. This is accomplished using ISL links. Cascaded FICON introduces the open systems SAN
concept of the Inter-Switch Links (ISLs). IBM now supports the flow of traffic from the processor through two
FICON directors connected through an ISL and on to the peripheral devices such as DASD and tape.
The majority of long distance connectivity is between two sites but it is becoming more prevalent to create a
number of remote locations to geographically distribute the computer processing elements to overcome a
disaster overtaking one of the data centers.
Below is an example of Brocade Blades with Long Wave Optics and Dark Fiber Connectivity for Applications
and RDR. The connectivity requirements are as follows:
Dark Fiber connections between two data centers
Dark Fiber can be either owned or leased by the customer

FIGURE 12 FC long distance cascading
The solution involves the following:
Use the proper Long Wave Length SFP
Use CWDM/DWDM to optimize fibers
Provision a proper number of buffer credits
Disaster Recover and Business Continuance reduces RPO and RTO
FCIP Trunking
FCIP Trunking is a Brocade innovation and exclusive technology that was originally developed for the
mainframe. Its primary purpose is to provide scalability and resiliency across multiple FCIP connections while
preventing IFCC. FCIP Trunking aggregates BW across multiple FCIP circuits forming a single logical ISL
between two VE_Ports.FCIP Trunking performs load balancing with failover/failback if a circuit goes offline. If
data in-flight is lost when a circuit goes offline that data is retransmitted across remaining links, and put back
into order prior to being sent to the Upper Layer Protocols (ULP, i.e. FICON or FCP). All data is always delivered
in-order. There is no requirement to enable IOD to get in-order delivery with FCIP Trunking.
FCIP trunking characteristics:
FCIP Trunking is an FCIP Tunnel with more than 1 circuit
Commonly known as a Link Aggregation Group (LAG)
FCIP Trunking provides the following benefits:
Single logical ISL in routing table as a single link, ULP sees a single link
Bandwidth Aggregation of each circuit
Load Balancing per batch across the available circuits
Failover to remaining circuits when a circuit is lost
Lossless Link Loss (LLL)
- Data in-flight is not lost when a link goes down, it will be retransmitted
- In-Order-Delivery (IOD) Does not require switch IOD to be enabled
- Data in-flight will be delivered in the correct order, even after a data loss in-flight
Works with both FICON and FC
- Supports FastWrite, OSTP, and FICON emulation over multiple circuits
Protocol Intermix Mode
Protocol Intermix Mode (PIM), which is also referred to as FICON/FCP intermix, is supported by IBM on
mainframes to enable FICON and open systems Fibre Channel Protocol (FCP) traffic to co-exist on the same
physical storage network.
When operating in a Protocol Intermix Mode (PIM) environment it is a good practice to not only zone FICON
ports away from FCP ports but also to control any currently unused mainframe ports by prohibiting them
using PDCMs or port disables. Ports that are prohibited will still have light but will not allow connectivity to any
other port. You could also block the port, turning off the light, which also disallows connectivity. When you
want to connect up an unconnected FICON port then you can go to the Allow/Prohibit Matrix and allow that
port to have connectivity (or unblock it).
Linux on System z and Storage Array replication through FICON switching devices are arguably the two top
reasons a customer might deploy a PIM environment.
FICON/FCP intermix can provide cost reductions such as the consolidation and management of both
switching infrastructure and cabling plants as well as access to the latest advances in technology, such as the
N_Port ID Virtualization (NPIV).
Examples for intermix:
- Small FICON and open systems environments with a common storage network
- z/OS or System z servers accessing remote FICON storage through FICON cascading
- Linux on a mainframe running z/OS using FICON to access local storage
- Hardware-based remote DASD mirroring between two locations using FCP as a transport, for example,
PPRC
- A mainframe processor that accesses storage via FICON and storage devices via FCP to perform
remote mirroring between devices (instead of ESCON)
- Open systems servers accessing storage on the same storage network using FCP
- Linux on the zSeries located at either of two sites using FCP to access storage
NOTE
Open systems use discovery; FICON uses IOCP
FICON and FCP
A FICON channel in Fibre Channel Protocol mode (which is CHPID type FCP) can access FCP devices as
follows:
- From a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a
SCSI device
- From a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a
Fibre Channel-to-SCSI bridge
FCP support enables Linux on System z to access industry-standard SCSI devices
FICON Express4, FICON Express2, and FICON Express channels in FCP mode provide full fabric
attachment of SCSI devices to the operating system images, using the Fibre Channel Protocol, and
provide point-to-point attachment of SCSI devices
For disk applications, these FCP storage devices use Fixed Block (512-byte) sectors instead of Extended
Count Key Data (ECKD) format used with FICON storage devices
If you are considering creating a disaster recovery site by synchronously replicating data between DASD at the
two sites, you would need to use FCP protocols between the DASD but still use cascaded FICON Directors to
manage the data flow. In this case, you need an intermix environment of FICON and FCP.
The FCP channel full fabric support enables switches and Directors to be supported between the System z
server and SCSI device, which means many hops through a storage area network
FICON channels in FCP mode can use the Queued Direct Input/Output (QDIO) architecture for
communication with the operating system
FCP channels do not use control devices but instead, data devices that represent QDIO queue pairs are
defined. These queue pairs consist of a request queue and a response queue. Each queue pair forms a
communication path between an operating system and the FCP channel. The enables an operating system to
send FCP requests to the FCP channel via the request queue. The FCP channel then uses the response queue
to pass a completion message and unsolicited status messages to the operating system.
HCD/IOCP is used to define the FCP channel type and QDIO data devices. There is no definition requirement
for the Fibre Channel storage controllers and devices in IOCP, nor the Fibre Channel devices such as switches
and Directors because of QDIO.
The FCP industry-standard architecture specifies that the Fibre Channel devices (end nodes) in a Fibre
Channel network use using World Wide Names (WWNs), Fibre Channel Identifiers (IDs), and Logical Unit
Numbers (LUNs) for addressing. These addresses are configured at the operating system level, and passed to
the FCP channel together with the corresponding Fibre Channel I/O via a logical QDIO device (queue).
Virtual Fabrics and FICON
Supported Features
Multiple Logical Switches (up to 8 on a DCX)
Can utilize 384 ports on a DCX chassis
Utilizes 48-port blades
Two instances of CUP (one CUP license per logical switch)
Not on base switch
Only supported in Native and Interopmode 2
Cloud Environments
Inter-chassis Links (ICLs)
If thought is being given to using some sort of cloud infrastructure, there are two technologies that could be of
assistance in a FICON environment. The first is the use of ICLs on a Brocade DCX.
FIGURE 13 Brocade DCX with ICLs
Each ICL is equivalent to 16 standard ISL ports (at 8 Gbps each) but take up no user ports
ICL cables can be a max of 2m in length
ICLs are silent hops; IBM does not count them as an official hop for FICON cascading. By using ICLs,
you can use four DCXs in a cascaded environment that does not violate the one hop rule.
Cascading of Directors and switches is limited to one hop for a FICON environment with the following
exceptions:
- The DCX Backbone with ICLs consists of 2 or 3 domains, but the hop across the ICLs can be
disregarded
- Because of this DCX<->ICL<->DCX<->ISL<->DCX<->ICL<->DCX is considered one hop
- FICON is supported for 16, 32 and 48 port blades
- 48-port blades on a DCX require the use of Virtual Fabrics
ICL Trunking on the Brocade DCX 8510-8 and 8510-4
The Brocade DCX 8510-8 has 4 port groups on the CR16-8 core blade. The Brocade DCX 8510-4 has 2 port
groups on the CR16-4 core blade. Each port group has 4 QSFP connectors, and each QSFP connector maps to
4 user ports.
Each of the 4 user ports in a QSFP terminates on a different ASIC, so a trunk cannot be formed among these
ports.
To establish ICL trunking between platforms in the Brocade DCX 8510 family, follow these configuration rules:
You need at least 2 ICLs between the platforms. A single ICL does not enable trunking
Each QSFP has four ports. However, these ports cannot form a trunk with each other, but can form trunks
only with corresponding ports on another QSFP.
You can have a maximum of 4 ports in an ICL trunk
You can have a maximum of 8 4-port trunks to a neighboring domain. Each core blade can have a
maximum of 4 ICLs to a neighboring domain
The QSFP cables must be connected to the same trunk group on each platform
N_Port ID Virtualization (NPIV)
NPIV is designed to allow the sharing of a single physical FCP channel among operating system images,
whether in logical partitions or as z/VM guests in virtual machines. This is achieved by assigning a unique
World Wide Port Name (WWPN) to each operating system connected to the FCP channel. In turn, each
operating system appears to have its own distinct WWPN in the SAN environment, thus enabling separation of
the associated FCP traffic on the channel.
Access controls based on the assigned WWPN can be applied in the SAN environment using standard
mechanisms such as zoning in FC switches and Logical Unit Number (LUN) masking in the storage controllers.
NPIV is exclusive to the System z servers and is the preferred method for sharing SCSI devices among multiple
operating system images when using a single physical FCP channel.
Benefits of Switched FICON
Minimize the number of CHPID ports required for connection to storage.
Switched FICON allows for simpler infrastructure management, lowered infrastructure cost of ownership,
and higher data availability.
Keep port, cable, or HBA failures to a single side of the system rather than allowing a complete connection
to be disrupted.
Can utilize Fan In Fan Out ratios to continue to scale new or existing mainframe applications even if all
the storage ports are in use.
Distribute workload more evenly and predictably across all of the available storage ports through a well
thought out Fan In Fan Out ratio
Ability to use NPIV with Linux
Can provide very robust, dynamic channel path connectivity providing ease of growth and scalability
throughout the FICON environment.
Ability to implement Control Unit Port (CUP) to obtain information such as buffer credit usage and other
performance statistics.
Allow DASD arrays to contain maximum capacity that can actually be utilized due to the Fan In Fan Out
architecture and balanced use of the storage ports.
Use FICON as a distance extension to support disaster recovery between two sites.
Minimize Mainframe Channel Card Costs with Switched FICON
Each storage port in a point-to-point connection requires its very own physical port connection on the
mainframe
In a cascaded FICON environment, the fan-in/fan-out ratios solve this problem just like it solves other
connectivity and scalability problems
- Fan-in: Attaching servers to storage via connected server/switch ports to a single storage subsystem
port
- Fan-out: Attaching multiple storage control units to a single server/switch
Every storage box has a maximum number of FICON connections:
- Tape might be 1 or 2
- Disk might be 2 to 48
In a cascaded FICON environment, you can use fan-in/fan-out ratios
- Use all storage ports
- Add more capacity to the DASD
Use CHPID fan-in to scale mainframe connectivity to the existing storage ports
Director/switch ports are less expensive than mainframe channel cards
NPIV can also expand the capabilities
Can get buffer credit usage into RMF
Remote Data Replication
Remote data replication is the process of creating replicas of information assets at remote sites (locations).
Remote replicas help organizations mitigate the risks associated with regionally driven outages resulting from
natural or human-made disasters. Similar to local replicas, they can also be used for other business
operations. The infrastructure on which information assets are stored at the primary site is called the source.
The infrastructure on which the replica is stored at the remote site is referred to as the target. Hosts that
access the source or target are referred to as source hosts or target hosts, respectively.
Two basic modes of remote replications are: Synchronous and Asynchronous replication. Data has to be
transferred from the source site to a target site over some network. This can be done over IP networks, over
the SAN, using DWDM (Dense Wave Division Multiplexing) or SONET (Synchronous Optical Network).
The following are types of data replication technologies:
Host based implies that all the replication is done by using the CPU resources of the host using software
that is running on the host.
- Logical Volume Manager (LVM) based
- Supports both synchronous and asynchronous mode
- Log Shipping
Storage Array based implies that all replication is done between Storage Arrays and is handled by the
Array Operating Environment.
- Support both synchronous and asynchronous mode
- Disk Buffered - Consistent Point-in-Time (PIT) copies
- Combination of Local and Remote Replication
Synchronous Data Replication
Data is committed at both the source site and the target site before the write is acknowledged to the host. Any
write to the source must be transmitted to and acknowledged by the target before signaling a write complete
to the host. Additional writes cannot occur until each preceding write has been completed and acknowledged.
It ensures that data at both sites are identical at all times.
Asynchronous Data Replication
In asynchronous data replication a write is committed to the source and immediately acknowledged to the
host. Data is buffered at the source and transmitted to the remote site later. Data at the remote site will be
behind the source by at least the size of the buffer. Hence, asynchronous remote replication provides a finite
(nonzero) RPO disaster recovery solution. RPO depends on the size of the buffer, available network
bandwidth, and the write workload to the source. There is no impact on application response time, as the
writes are acknowledged immediately to the source host. This enables deployment of asynchronous
replication over extended distances. Asynchronous remote replication can be deployed over distances ranging
from several hundred to several thousand kilometers between two sites.
FICON Emulation Overview
FICON emulation supports FICON traffic over IP WANs using FCIP as the underlying protocol. FICON emulation
can be extended to support performance enhancements for specific applications through use of the following
licensed features:
IBM z/OS Global Mirror (formerly eXtended Remote Copy or XRC)
FICON Tape Emulation (tape read and write pipelining)
The 4Gbps FR4-18i blade uses the following licenses:
High-Performance Extension over FCIP/FC
Allows creation of tunnels.
FICON Tape
Allows FICON Tape emulation.
IBM z/OS Global Mirror
Allows FICON IBM z/OS Global Mirror emulation.
Brocade Accelerator for FICON
Enables both FICON tape emulation and IBM z/OS Global Mirror emulation. One license must be used per
unit or blade.
Bandwidth Allocation and Restrictions
Multiple FCIP tunnels are not supported between pairs of 7800 switches or FX8-24 blades when any of
the FICON emulation/acceleration features or FCP acceleration features are enabled on the tunnel,
unless TI Zones or LS/LF configurations are used to provide deterministic flows between the switches.
VE_Port 12 has two 10 Gbps circuits. Circuit 0 has a metric of 0 on xge1 and circuit 1 has a metric 1 on
xge0. With this configuration, no other tunnels or circuits would be allowed on this blade because both
XGE ports have 10 Gbps of configured bandwidth.
A single VE_Port is required on each end.
3 Implementation
After completing this section you should be able to perform the following:
Describe basic FICON installation best practices
Demonstrate the ability to implement a cascaded FICON solution in a pure Fabric OS environment
Best Practices
Here are some reasons why you should always enable FMS on your FICON switching devices. As stated, it is a
best practice to also implement CUP on your FICON switching devices:
RMF can have in-band access to the FICON switching.
Systems Automation for z/OS, with the I/O Operations IOOPs module implemented, can have in-band
access to the FICON switching devices.
FICON Dynamic Channel Management (DCM) can dynamically add and remove channel resources at the
Workload Managers discretion.
When CUP is enabled, z/OS will use Coordinated Universal Time (UTC) to update the time/date on all of
the CUP-enabled switching devices. z/OS keeps everything in sync and all logs are also synchronized.
The Allow/Prohibit Addressing Matrix becomes available
The Active=Saved condition becomes user configurable
Zoning
In a single director fabric with just FICON, it is a common practice to place all ports in a single specific
zone.
If supporting multiple operating systems from LPARs on the same director, such as z/OS and z/TPF, create
separate zones for each operating system from the LPAR.
In intermixed environments, place the FICON ports in a specific FICON port zone. Then use standard open
systems best practices for creating World Wide Name (WWN) zones for the FCP traffic on that same
chassis.
It is a bad practice to mix FICON and FCP ports within the same zone.
In cascaded environments, paths using 2-byte addressing and their remote devices should be placed in
zones that are separate from non-cascaded paths.
Do not put the E_Ports (ISL ports) in a zone. This can have either a default zone or specific zones, but not
both on the same chassis.
Do not use the Default Zoning Mode (all access) for FICON, but rather define port zones (domain/index)
for FICON and WWN zones for FCP if using protocol intermix.
Brocade generally recommends that customers simply place all FICON devices into one large zone, which
achieves the same behavior as all access, but provides the added protection that any new device is not
allowed to talk to any other device until it is explicitly added to the zone.
By disabling the default zone (no access) and activating port zones, you ensure that if you utilize Prohibit
Dynamic Connectivity Matrix (PDCM) entries that they are honored and enforced.
Zoning RSCN cannot be suppressed or ignored on B-Series chassis.
FOS v7.0 and higher allows a switch using a Default Zoning Mode of (no access), and with no zoning
configuration, to merge with a fabric that has an active zone configuration
Domain ID
Always make domain IDs unique and insistent. Insistent domain IDs are required for 2-byte addressing
(cascading).
There is no need for a domain ID to ever change in a FICON environment; fabrics come up faster after
recovering from a failure if the domain ID is insistent. Each switch needs its own Switch ID in the IOCP so,
assuming you are making the Switch ID = Switch Address, each domain ID must be unique. Note that the
switch address is based on the domain ID.
This allows the directors to be cascaded in the future without having to take any director off line.
Make sure that the domain offset (normally x60) is the same on all of the devices in all of your FICON
fabrics.
Switch ID
Set the switch ID in the IOCP to be the same as the switch address (on Brocade DCX switches this is also
the Domain ID) of the FICON switching device.
In Interop Mode 0 (this is the typical mode and is used for fabrics with B-type switches only) the switch
address and the domain ID are the same thing.
In Interop Mode 2 (this is the mode for intermixing M-Series and B-Series directors in the same fabric), the
M6140 uses an offset of 96 (0x60). For example, in Interop Mode 2, a domain ID of 6 would have a switch
address of 0x66; therefore, the switch ID should be 66. Please keep this in mind when configuring any
future domain IDs on a B-Series switch.
Note that channel statements in the IOCP are associated with a switch ID; however, the link statements,
when using 2-byte addressing, are the switch address. This avoids any confusion as to which switch you
are using.
Fill Words
Prior to 8 Gbps, the fill word was always Idle.
With 8 Gbps, copper wire EMI emissions were too high; using ARBs instead of Idles helps to lower
emissions. This really had no effect on fiber cablesjust copper cables.
The FC specifications were maturing at the time that Brocade, and other vendors, implemented the fill
word. The original FC-PI (Physical Interface) specification did not articulate what to do during initialization:
- As a result, different vendors implemented different solutions.
- IBM ships 8 Gbps defaulted to mode 1 (ARBs).
- There are other OEMs that ship 8 Gbps defaulted to mode 0 (Idles).
At Brocade FOS 6.3.1 and above, Brocade offers four mode settings.
Use the portCfgFillWord command to set the mode:
- Mode 0: Idle/Idle
- Mode 1: Arb/Arb
- Mode 2: Idle/Arb (compliant with current spec)
- Mode 3: Try mode 1 first, then try mode 2
Example: The portcfgfillword 1/8, 0 command changes the fill word for Slot1, Port8 to
Idles.
Example: The portcfgfillword 1/8, 1 command changes the fill word for Slot1, Port8 to
ARB(FF).
HDS storage supports only mode 0 or 2. HDS recommends mode 2 for their 8 Gbps devices.
Obviously, if a vendor asks for a specific mode, you should try that mode.
It is important to note that the portcfgfillword command is only for 8 Gbps switching devices and is
not supported on any 16 Gbps port card, regardless of the SFP installed (8 or 16 Gbps). The fill word still
needs to be set on 8 Gbps products (Condor2 ASIC). This includes ports on the Brocade FX8-24 Extension
Blade even if it is installed in a Brocade DCX 8510.
Traffic Isolation Zoning Overview
The Traffic Isolation Zoning feature allows you to control the flow of interswitch traffic by creating a dedicated
path for traffic flowing from a specific set of source ports (N_Ports). For example, you might use Traffic
Isolation Zoning for the following scenarios:
To dedicate an ISL to high priority, host-to-target traffic.
To force high volume, low priority traffic onto a given ISL to limit the effect on the fabric of this high traffic
pattern.
To ensure that requests and responses of FCIP-based applications such as tape pipelining use the same
VE_Port tunnel across a metaSAN.
TI Zone Failover
A TI zone can have failover enabled or disabled.
Disable failover if you want to guarantee that TI zone traffic uses only the dedicated path, and that no other
traffic can use the dedicated path.
Enable failover if you want traffic to have alternate routes if either the dedicated or non-dedicated paths
cannot be used.
Illegal ETIZ Configuration: Separate Paths from a Port to Devices on Same Domain
Figure 41 shows two enhanced TI zones that are configured incorrectly because there are two paths from a
local port (port 8 on Domain 3) to two or more devices on the same remote domain (ports 1 and 4 on Domain
1).
The TI zones are enhanced TI zones because they have an overlapping member (3,8). Each zone describes a
different path from the Target to Domain 1. Traffic is routed correctly from Host 1 and Host 2 to the Target;
however, traffic from the Target to the Hosts might not be.
Traffic from (3,8) destined for Domain 1 cannot go through both port 6 and port 7, so only one port is chosen.
If port 6 is chosen, frames destined for (1,4) will be dropped at Domain 1. If port 7 is chosen, frames destined
for (1,1) will be dropped.
FIGURE 14 Illegal ETIZ configuration: two paths from one port to two devices on the same remote domain
256-area Addressing Mode
This configurable addressing mode is available only in a logical switch on the Brocade DCX, DCX-4S, and
Brocade DCX 8510 family enterprise-class platforms. In this mode, only 256 ports are supported and each
port receives a unique 8-bit area address. This mode can be used in FICON environments, which have strict
requirements for 8-bit area FC addresses.
There are two types of area assignment modes in the 256-area addressing mode: zero-based and port-based.
Zero-based mode, which assigns areas as ports, are added to the partition, beginning at area 0x00. This
mode allows FICON customers to make use of the upper ports of a 48-port or 64-port blade. Zero-based
mode is also supported on the default switch.
Port-based mode does not support the upper 16 ports of a 48-port or 64-port blade in a logical switch.
Port-based mode is not supported on the default switch.
Port Swapping
If a port malfunctions, or if you want to connect to different devices without having to re-wire your
infrastructure, you can move a ports traffic to another port (swap ports) without changing the I/O
Configuration Data Set (IOCDS) on the mainframe computer.
The following are port swapping considerations:
Ports with area OxFE or OxFF addresses cannot be swapped when FMS mode is enabled.
GbE ports cannot be swapped
Shared area ports cannot be swapped.
Ports that are part of a trunk group cannot be swapped.
Swapping ports between different logical switches is not supported. The ports on the source
and destination blades need to be in the same logical switch.
Geographically Dispersed Parallel Sysplex
Geographically Dispersed Parallel Sysplex (GDPS) Manages System z application and data availability at both
local and remote sites. It has the following capabilities:
Monitors systems, disk and tape subsystems
Manages planned and unplanned activities
System/disk maintenance/failure
Site maintenance/failure
Clustering
Remote copy (disk and tape)
Automation
Easy to use interface
Intuitive dashboard
Policy based commands
Established and experienced
Over 10 years of enhancement
Implementation success
Supported worldwide
There are different vehicles that constitutes the GDPS service such as HyperSwap Manager, the manager for
Metro Global Mirroring, as well as z/OS Global Mirror, formerly known as XRC. These are supported as well as
two and three site solutions.
Troubleshooting Trunking
The trunkDebug command displays the possible reason that two ports cannot be trunked. Possible reasons
are:
The switch does not support trunking.
A trunking license is required.
Trunking is not supported in switch interoperability mode.
Port trunking is disabled.
The port is not an E_Port.
The port is not 2 Gbps, 4 Gbps, or 8 Gbps.
The port connects to a switch other than the one you want it to.
To correct this issue, connect additional ISLs to the switch with which you want to communicate.
The ports are not the same speed or they are not set to an invalid speed.
Manually set port speeds to a speed supported on both sides of the trunk.
The ports are not set to the same long distance mode.
Set the long distance mode to the same setting on all ports on both sides of the trunk.
Local or remote ports are not in the same port group.
Move all ISLs to the same port group. The port groups begin at port 0 and are in groups of 4 or 8,
depending on the switch model. Until this is done, the ISLs do not trunk.
The difference in the cable length among trunked links is greater than the allowed difference.
Native Connectivity (M-EOS interoperability)
A switch running FOS v7.0 cannot form E-port connectivity with any M-EOS platform. A switch running FOS v7.0
can only operate in Brocade native mode (interopmode 0). Connectivity between M-EOS platforms and a
switch running FOS v7.0 is supported via FCR.
ISL Decommission
The Port Decommission feature provides users with the ability to non-disruptively remove an ISL from service.
When an ISL is selected for decommissioning, the switches communicate with each other to coordinate the
movement of flows off of the ISL to alternative paths. Once the flows are moved to alternative paths, the
switches block the E_Ports associated with the ISL being decommissioned to complete the decommission
process.
At FOS 6.4.0a + (2010), DLS can and should always be utilized with FICON. Lossless DLS pauses ingress ports
before moving traffic routes. This capability provides customers with the ability to enable DLS for FICON.
The term Lossless describes a behavior that FOS uses when moving I/O traffic between the parallel ISL link
routes.
4 Management
After completing this section you should be able to perform the following:
Demonstrate knowledge of how to use Brocade tools to manage FICON solutions
Describe the benefits and uses of FICON CUP
Demonstrate knowledge of firmware and configuration management tools
Command Line Tools
Diagnostic Port
Fabric OS v7.0.0 allows you to convert a fibre channel port, including ISLs and loopback ports, into a
Diagnostic Port (D_Port). This port lets you isolate the inter-switch link (ISL) to diagnose link level faults. The
D_Port does not carry any fabric traffic, and is designated to run only specific diagnostics tests on it. The
creation of a D_Port is subject to Virtual Fabric restrictions that may be in place. The ports must be 10G or
16G Brocade-branded SFPs on a Brocade DCX 8510, and running Fabric OS v7.0.0 or later.
Pathinfo
The pathinfo command displays routing and statistical information from a source port index on the local
switch to a destination port index on another switch. This routing information describes the full path that a
data stream travels between these ports, including all intermediate switches.
Blade Swapping
Blade-based port swap is mainly used for FICON and is only applicable for port blades. However, the
Management application does not block blade-based port swap for other application blades, including the 8
Gbps 24-port blade.
You can swap all of the ports from one blade to another blade. During this operation all ports in the selected
blades are swapped. This operation disrupts the traffic on all ports for the selected blades. If GbE ports are
present on the blade, only the non-GbE ports are swapped. To swap blades, you must meet the following
requirements:
The chassis must be running Fabric OS 6.3 or later.
The chassis must have at least two blades of same type present.
To perform the Swap Blades function you must have Read and Write access for the Product Administration
privilege.
Web Tools - Performance Monitoring
Web Tools predefines basic graph types, to simplify performance monitoring. A wide range of end-to-end
fabric, LUN, device, and port metrics graphs are included.
The advanced monitoring graphs give more detailed performance information to help you manage your
fabric
You can access the basic monitoring graphs on all switches
Advanced Monitoring graphs are available only on switches that have a Brocade Advanced Performance
Monitoring license activated
Graphs:
- Port Throughput
The performance of a port, in bytes per second, for frames received and transmitted
- Switch Aggregate Throughput
The aggregate performance of all ports on a switch
- Blade Aggregate Throughput
The aggregate performance of all ports on a port card. This graph is available only for the Brocade
48000 and Brocade DCX Backbone.
- Switch Throughput Utilization
The port throughput, in Gbps, at the time the sample is taken
- Port Error
CRC errors for a given port
- Switch Percent Utilization
The percentage utilization for each port in a switch
- Port Snapshot Error
The CRC error count between sampling periods for all the ports on a switch
SAN Health
IOCP
SAN Health is compatible with FICON switches and can be used to review IOCP data with the SAN
environment.
The IOCP file contains the path configuration statements for a given mainframe system.
- SAN Health will match these path statements with the physical switch ports found in the SAN Health
audit
While SAN Health is run, all interaction with the switches is displayed and can be stopped at any time.
The diagnostics capture completes in approximately two minutes.
SAN Health will remain open and capture performance data for the duration that you have set the
performance capture
FIGURE 15 SAN Health Output with IOCP Data
To the standard SAN Health Report and Visio diagram, Brocade adds FICON information including the Serial/
Sequence number. This information appears anywhere where there is port specific information in the report.
AN EXcel spreadsheet is also produced with CHPIDs and CNTLUNITs.
Types of Bottlenecks
The bottleneck detection feature detects two types of bottlenecks:
Latency bottleneck
Congestion bottleneck
A latency bottleneck is a port where the offered load exceeds the rate at which the other end of the link can
continuously accept traffic, but does not exceed the physical capacity of the link. This condition can be
caused by a device attached to the fabric that is slow to process received frames and send back credit
returns. A latency bottleneck due to such a device can spread through the fabric and can slow down unrelated
flows that share links with the slow flow.
By default, bottleneck detection detects latency bottlenecks that are severe enough that they cause 98% loss
of throughput. This default value can be modified to a different percentage.
A congestion bottleneck is a port that is unable to transmit frames at the offered rate because the offered
rate is greater than the physical data rate of the line. For example, this condition can be caused by trying to
transfer data at 8 Gbps over a 4 Gbps ISL.
You can use the bottleneckMon command to configure alert thresholds for congestion and latency
bottlenecks.
FICON CUP Zoning and PDCM Considerations
The FICON PDCMs control whether or not communication between a pairs ports in the switch is prohibited or
allowed. PDCMs are a per-port construct. Each port on the switch has its own PDCM that defines whether
communication is allowed between that particular port, and each of the other ports in the switch, including
itself. The Allow/Prohibit Matrix presents a matrix that allows you to set and clear the PDCMs for all external
ports. If there are any differences between restrictions set up with Brocade Advanced Zoning and PDCMs, the
most restrictive rules are automatically applied.
FMS must be enabled for the Allow/Prohibit matrix to be operational, and there must be an active zoning
configuration. (D,I) zoning is recommended for FICON, but is not specifically required.
Fabric Watch Overview
Fabric Watch is an optional storage area network (SAN) health monitor that allows you to enable each switch
to constantly monitor its SAN fabric for potential faults and automatically alerts you to problems long before
they become costly failures.
Reasons to Customize Fabric Watch Settings
Customization is recommended to achieve the following objectives:
Selecting one or more event settings
Selecting an appropriate message delivery method for critical and noncritical events
Selecting appropriate thresholds and alarm levels relevant to each class element
Defining the appropriate Time Base event triggering based on the class element traits
Eliminating message delivery that has little or no practical value to the SAN administrator
Consolidating multiple messages generated from a single event
Fabric Monitoring Setting Guidelines
It is recommended that you leave the entire Fabric class in its default state (no alerts) for the following
reasons:
Domain ID changes
Plan and use strict change control practices to avoid Domain ID changes.
Loss of E_Port
Detect if an E_Port is down using the E_Port class areas.
Fabric logins
In a large environment of numerous devices, this area is of no interest.
Fabric reconfiguration
Fabric reconfigurations typically occur when new switches are added to a fabric, which is a planned
activity, or when an upstream or downstream ISL fails, which is detected through the E_Port class areas.
Since fabric reconfigurations are monitored elsewhere, dont change the default settings for the Fabric
class.
Segmentation changes
Segmentations only occur in the event of an entire switch failure. In this rare case, you can gather multiple
reports from all the attached E_Ports of the link failures.
Zoning changes
Zone changes are captured through the Audit facility in Fabric OS. All zone changes can be configured to
be recorded in the RASlog, which is the recommended practice.
Use the portcfgautodisable command to enable or disable the autodisable feature for a specified port
or a range of ports and to manage the configuration.
The port autodisable feature minimizes traffic disruption introduced in some instances of automatic port
recovery. When the autodisable flag is set, you can specify the conditions that will prevent the port to
reinitialize. Such conditions include loss of sync, loss of signal, OLS, NOS, and LIP.
Advanced Performance Monitoring Overview
Based on Brocade Frame Filtering technology and a unique performance counter engine
Comprehensive tool for monitoring the performance of networked storage resources
Advanced Performance Monitoring provides the following monitors
End-to-end monitors measure the traffic between a host/target pair
Filter-based monitors measure the traffic transmitted through a port with specific values in the first 64
bytes of the frame
ISL monitors measure the traffic transmitted through an InterSwitch Link (ISL) to different destination
domains
Top Talker monitors measure the flows that are major consumers of bandwidth on a port
The Top Talker (TT) feature is an enhancement to APM end-to-end monitors
- First available in Fabric OS v6.0
- When enabled, these monitors determine which SID-DID pairs are the major users of switch F_Port
bandwidth
- Can be enabled on specific switch F_Ports on any switch in the fabric
RMF 74-7 Records
Enabling RMF 74 subtype 7 (RMF 74-7) records yields an RMF report called the FICON Director Activity
Report
Data is collected for each RMF interval if FCD is specified in the ERBRMFnn parmlib member. The
report captures information based on an interval which is set when you create the report.
- The report captures a snapshot of data and the counters based on an time interval, such as 60
seconds. Often, you need to run these reports more than once and change the interval periods for
troubleshooting to determine if there is a trend.
- FMS must be enabled on the switch
- IECIOSxx dataset SYS1.PARMLIB must specify STATS=YES
This RMF report is often overlooked but contains very meaningful data concerning FICON I/O
performance - in particular, frame pacing delay
Frame pacing delay is the only available method to indicate a BB_Credit starvation issue on a given port
FIGURE 16 FICON Director Activity Report
RMF FICON Director Activity Report
FIGURE 17 FICON Director Activity Report for an M6140
The two values contained in the report that are most often used for troubleshooting are:
- AVG FRAME PACING refers to frame pacing delay. If it is a non-zero number then it reflects the number
of times that I/O was delayed for 2.5 microseconds or longer due to buffer credits falling to zero.
- ERROR COUNT which is the number of errors which were encountered during the interval.
The AVG FRAME PACING column is where frame pacing delay is reported.
Even if the number is zero there could be a buffer credit shortage problem:
- Switching device is not fast enough to catch all errors (M6140, Mi10K, etc.)
- Buffer credits went to zero but delays to send the next frame were less than 2.5 microseconds
Sampling times can be an issue
IFCCs
The System z FICON channels approach to capturing error information at the time of a FICON I/O
operation failure is to detect an Interface Control Check (IFCC). The FICON purge-path-extended function
requests, collects, and transfers link error statistical buffer information from each Fibre Channel port in
the FICON channel-to-CU path to the System z host (MVS) console.
The FICON purge-path-extended function is used by the mainframe to issue a query to determine the end-
to-end path information
Example of an IFCC Error Message: IOS051I
- 2027-20:17:04 (MVS 20:16:54) IOS051I INTERFACE TIMEOUT DETECTED ON
D829,D0,E7,**02,PCHID=0520
- A possible MIHPTO issue
The message help provides this:
- IOS051I INTERFACE TIMEOUT DETECTED ON dev, chp, cmd, stat
- Explanation: The channel subsystem detected a timeout condition during the operation of device. This
message is issued as an informational message only for the interface timeouts that occur on native
FICON channels
- In the message text:
dev (D829)is the device number.
chp (D0) are the channel path identifiers (CHPID), if known, otherwise, this field is set to
asterisks
cmd(E7) is the failing command code, if known; otherwise, this field is set to asterisks
stat(**02) is the device and subchannel status, if known; otherwise, this field is set to asterisks.
In the example, the **02 indicates an IFCC which indicates a problem with the device.
PCHID=0520 is the physical channel path identifier
Saving or Copying Allow/Prohibit Matrix Configurations to Another Device
When copying or saving a configuration from a small switch (source switch with fewer ports; for example, 64
ports) to a larger switch (destination switch with a larger number of ports; for example, 256 ports) only the
port address range of the smaller switch will be affected on the larger switch. All additional port addresses will
display the default settings (port state defaults to Allow and the Blocked check box defaults to not checked).
Copying or saving a configuration from a larger switch to a smaller device only copies or saves the port
address range that matches the smaller switch. Additionally a message displays that the additional port
addresses from the larger switch are discarded.
When copying or saving a configuration from or to Logical Switches, the only ports affected are the port
addresses defined in the Logical Switch. The FICON CUP daemon retains the full compliment of records
regardless of the size of the Logical Switch. Therefore, copying or saving a configuration from or to logical
switches should work the same as copying or saving between standard switches.
Cascaded FICON fabric
NOTE
You must have FICON Management privileges to configure a fabric for cascaded FICON.
The FICON Configuration Management application in Brocade Network Advisor enables you to easily configure
a fabric for cascaded FICON. Note that configuring a fabric for cascaded FICON may be disruptive to current I/
O operations in the fabric, as it needs to disable and enable the switches in the fabric.
FICON configuration performs the following operations on the selected fabric:
Turns on the insistent domain ID flag (IDID) on all switches.
Sets High Integrity Fabric Configuration (HIFC) on the seed switch.
NOTE
High Integrity Fabric (HIF) activates SCC policy, sets Insistent Domain ID and sets Fabric Wide Consistency
Policy for SCC in tolerant mode.
- Fabric-wide consistency policy (FWCP) is configured to include SCC in strict mode.
- SCC policy is created or modified to limit connectivity to only the switches in the selected fabric.
Enables port-based routing on all switches.
Enables In-Order Delivery (IOD) on all switches.
Enables Dynamic Load Sharing (DLS) based on user selection and the firmware level.
NOTE
To enable DLS, all switches in the fabric must be 8 Gbps or faster and running Fabric OS 6.4 or later.
(Optional) Turns on FICON Management Server (FMS) mode on all switches.
- If switches are running Fabric OS 7.0 and later, FMS will not be enabled unless the switches have an
active CUP license.
- If switches are running Fabric OS earlier than 7.0 and not have a CUP license, after successful
configuration, you can access the Port Connectivity (Allow/Prohibit) matrix, but the host system
cannot communicate with the FICON Management Server unless you install a CUP license. If a CUP
license is later installed on these switches, then FMS mode must be re-enabled on these switches.
Firmware Downloads
The CUP device must be varied offline to all MVS partitions before starting a code load. The CUP device can be
varied back online after the code load completes. Failure to vary off the CUP devices may result in missing
interrupt.
FCIP
A disabled XGE or GE port will be re-enabled after a code upgrade/downgrade. To prevent XGE/GE ports from
being re-enabled after a code load they should be persistently disabled.
5 Maintenance
After completing this section you should be able to demonstrate knowledge of tools used for problem
determination.
Top Talker Monitors
Top Talker monitors determine the flows (SID/DID pairs) that are the major users of bandwidth (after initial
stabilization). Top Talker monitors measure bandwidth usage data in real-time and relative to the port on
which the monitor is installed.
NOTE
Top Talkers is a feature under the Advanced Performance Monitoring license.
Applications can use the Top Talker data to do the following:
Re-route the traffic through different ports that are less busy, so as not to overload a given port.
Alert you to the top-talking flows on a port if the total traffic on the port exceeds the acceptable bandwidth
consumption.
You can use Top Talkers to identify the SID/DID pairs that consume the most bandwidth and can then
configure them with certain Quality of Service (QoS) attributes so they get proper priority.
General Information to Gather for all Cases
The following information needs to be gathered for all FICON setups:
The output from the standard support commands (portLogDump, supportSave, supportShow) the Fabric
Manager Event Log, EFCM or Brocade Network Advisor logs.
By default, the FICON group in the supportShow output is disabled. To enable the capture of FICON data in
the supportShow output, enter the supportshowcfgenable ficon command. After you get
confirmation that the configuration has been updated, the following will be collected and appear in the
output for the supportShow command:
- ficonCupShow fmsmode
- ficonCupShow modereg
- ficonDbg dump rnid
- ficonDbg log
- ficonShow lirr
- ficonShow rlir
- ficonShow rnid
- ficonShow switchrnid
- ficuCmd dump -A
Type of mainframe involved. Need make, model, and driver levels in use.
Type of actual storage array installed. Many arrays will emulate a certain type of IBM array and we need to
know the exact make, model, and firmware of the array in use.
Other detailed information for protocol-specific problems:
- Port data structures, displayed using the ptdatashow command.
- Port registers, displayed using the ptregshow command.
Read Brocade Release Notes for specific version information regarding the Fabric OS installed.
Taking the Test
After the Introduction Screen, once you click on Next, you will see the following non-disclosure agreement:
IMPORTANT: PLEASE READ THE FOLLOWING BROCADE NON-DISCLOSURE CONFIDENTIALITY AGREEMENT
CAREFULLY BEFORE TAKING THIS EXAM.
The following Non-Disclosure Confidentiality Agreement (the Agreement) sets forth the terms and
conditions of your use of the exam materials as defined below.
The Disclosure to you of this Exam and any questions, answers, worksheets, computations, drawings,
diagrams, or any communications, including verbal communication by any party, regarding or related to
the Exam and such Exam Materials and any derivatives thereof is subject to the Terms and Conditions of
this Agreement.
You understand, acknowledge and agree:
That the questions and answers of the Exam are the exclusive and confidential property of Brocade
and are protected by Brocade intellectual property rights;
That you may not disclose the Exam questions or answers or discuss any of the content of the Exam
Materials with any person, without prior approval from Brocade;
Not to copy or attempt to make copies (written, photocopied, or otherwise) of any Exam Material,
including, without limitation, any Exam questions or answers;
Not to sell, license, distribute, or give away the Exam Materials, questions, or answers;
You have not purchased, solicited or used unauthorized (non-Brocade sanctioned) Exam Materials,
questions, or answers in preparation for this exam;
That your obligations under this Agreement shall continue in effect after the Exam and, if applicable,
after termination of your credential, regardless of the reason or reasons for terminations, and whether
such termination is voluntary or involuntary.
Brocade reserves the right to take all appropriate actions to remedy or prevent disclosure or misuse,
including, without limitation, obtaining an immediate injunction. Brocade reserves the right to validate all
results and take any appropriate actions as needed. Brocade also reserves the right to use any
technologies and methods for verifying the identity of candidates. Such technology may include, without
limitation, personally identifiable information, challenge questions, identification numbers, photographic
information, and other measures to protect against fraud and abuse.
Neither this Agreement nor any right granted hereunder shall be assignable or otherwise transferable by
you.
By clicking on the "A" button (YES, I AGREE), you are consenting to be bound by the terms and
conditions of this agreement and state that you have read this agreement carefully and you understand
and accept the obligations which it imposes without reservation. You further state that no promises or
representations have been made to induce agreement and that you accept this agreement voluntarily and
freely.
A. YES, I AGREE
B. NO, I DO NOT AGREE

BCAF 2013 in A Nutshell Study Guide For Exam 143-130: Brocade University Revision 0213

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

BCAF 2013 in A Nutshell Study Guide For Exam 143-130: Brocade University Revision 0213

Hochgeladen von

Copyright:

Verfügbare Formate

BCAF 2013 in a

Nutshell Study Guide

Das könnte Ihnen auch gefallen