Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812

Project Name: Date: 04 Apr 2011

Contract Number: Page: 1 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

This specification has been revised as indicated below and described in the revision record on the
following page. Please destroy all previous revisions.

Revision Date Originator's Reviewed/Checked By Pages

No. Name & Initials Name & Initials
0 04/04/2011 A. Touma Anna M. Pham 17

APPROVALS SIGNATURES DATE

Lead Engineer Thomas B. Robinson

Project Manager: Russell Shulz
Client Approval:

ISSUED FOR : Construction X Other Review/Approval

Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 2 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

Record of Revisions

Revision Date Description

No.

0 4/4/2011 Issued for Review/Approval

Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 3 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

TABLE OF CONTENTS

Section

1.0 GENERAL......................................................................................................................................................... 4

2.0 REFERENCES................................................................................................................................................. 4

3.0 ABREVIATION................................................................................................................................................. 4

4.0 SPECIFIC REQUIREMENTS ........................................................................................................................ 5

4.1 ARCHITECTURE.............................................................................................................................. 5
4.2 DESIGN PARAMETERS................................................................................................................. 6
4.3 NETWORK MANAGEMENT ........................................................................................................14
4.4 SECURITY .......................................................................................................................................15
5.0 SYSTEM ARCHITECTURE DIAGRAM ....................................................................................................17
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 4 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

1.0 GENERAL

A Storage Area Network (SAN) shall be implemented in accordance with the latest SNIA,
ANSI and FCIA standards on Fibre Channel technology. The SAN is any high-performance
network whose two primary purposes are to perform two major functions:

 Storage consolidation: Provide dynamic high speed connection between each

MARAFIQ enterprise hosts and storage devices.

 Disaster recovery: Store, maintain and transfer critical storage data mirrored
over to other MARAFIQ Locations.

This specification describes the architecture, design parameters, network management and
security requirements for the SAN.

2.0 REFERENCES

The SAN shall comply with this specification and applicable codes and standards. Should
conflicts arise, the Owner will be consulted but in general, the most stringent requirements will
be applied.

3.0 ABREVIATION

ANSI American National Standards Institute

BIC Building Inside Cabling
BER Bit Error Rate
CLAN Control LAN
FCIA Fibre Channel Industry Association
FCIP Fibre Channel over IP
FCP Fibre Channel Protocol
FSPF Fabric Shortest Path First
HBA Host Bus Adapter
HTTP Hype Text Transfer Protocol
IEEE Institute of Electrical and Electronics Engineers
IP Internet Protocol
ISL Inter-Switch Link
FSPF Fabric Shortest Path First
LAN Local Area Network
MARAFIQ Power and Water Utility Company for Jubail and Yanbu
M-M Multi-Mode
PCI Peripheral Component Interconnect
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 5 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

RAID Redundant Array of Independent Disks

S-M Single-Mode
SAN Storage Area Network
SNIA Storage Network Industry Association
SCSI Small Computer System Interface
SDH Synchronous Digital Hierarchy
SFP Small Form-Factor Pluggable
SNMP Simple Network Management Protocol
TCP Transport Control Protocol
WAN Wide Area Network
WWN World Wide Number
XFP 10 Gig Small Form-Factor Pluggable

4.0 SPECIFIC REQUIREMENTS

4.1 Architecture

4.1.1 General

A. The ultimate goal of the SAN is to move data quickly and reliably
between local or remote hosts and storage resources. This shall be
accomplished using a switched Fibre Channel network. Refer to section
titled “System Architecture Diagram”.

B. The switched Fibre Channel network consists of high speed switches,

scalable disk storage system, and the interconnecting components.

C. The topology for the Fibre Channel SAN shall be mesh configured for
redundancy using full duplex fabric switches connected to fiber optic
cables. For wide area connections, the topology is point-to-point using
fiber optic cabling.

4.1.2 Local SAN Operation

A. The local Fibre Channel SAN operates independent of other networks

(e.g. LAN, WAN, CLAN, etc.) defined for MARAFIQ.

B. The local SAN operates over a redundant 4 Gbps backbone that provides
extended storage space for LAN hosts.

C. Any LAN client may store and retrieve data from the MARAFIQ storage
clusters by either directly connecting to the SAN or via the enterprise
servers attached to the SAN. The storage system may consist of many
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 6 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

disk clusters. Some clusters may contain mirrored data from other
MARAFIQ sites.

4.1.3 Wide Area SAN-to-SAN Operation

A. The preferred wide area operation is to have the SAN bypass the SDH
WAN, directly linking the main MARAFIQ location with other MARAFIQ
locations across a Fibre Channel backbone. High speed 10 Gbps
transceivers shall allow direct long haul links up to a distance of 100km
away.

B. Throughputs for the SAN across the wide area shall be 10 Gbps or
higher to ensure rapid transfer of data to remote location during
emergencies.

C. The SAN shall have the option to interface with the SDH WAN via a link
that supports or converts FC protocols. For example, using a TCP/IP
multi-protocol router that connects with the SDH WAN allows the
transmission of Fibre Channel information to be tunneled through the IP
network by encapsulating the Fibre Channel data and subsequently
transporting it over a TCP socket. This method however increases the
likelihood of errors since additional processing of the original FC frame is
required.

D. Remote storage data from other MARAFIQ locations shall periodically be

mirrored over the wide area link to two distinct storage clusters in the
main MARAFIQ location.

E. The main MARAFIQ location storage cluster shall mirror over the wide
area link to a remote cluster available at another MARAFIQ Location.

4.2 Design Parameters

24
4.2.1 The local MARAFIQ SAN shall support up to 2 switched ports with each
port able to sustain throughput of 400MB/s (approximately 4 Gbps).

4.2.2 Physical System

A. Fabric Switches

1. Switches shall be capable of handling multilayer operating

system platforms. Refer to section titled “Storage Capacity” for a
list of required platform types.

2. Switches shall provide the following services for the SAN:

a. Management Services: communicates with Network

Management System.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 7 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

b. Time Services: synchronization with SAN devices.

c. Name Services: Maps WWN with 24 bit-port addresses

(refer to section titled “Addressing”).

d. Login Services: required by each SAN node with its

local fabric switch.

e. Registered State Change Notification: send information

about a change in the state of one node to all other
nodes in the fabric.

3. Switches shall provide the following services for the SAN:

4. Switches shall be rack mountable with dual cooling and power

supplies.

5. Non-disruptive software upgrades to the switch may be

performed locally or from a remote workstation.

6. The switches may be cascaded to increase capacity.

7. ISL’s may be grouped together to form an aggregate non-

blocking trunk between the switches.

8. The switches shall follow a mesh core-edge design. This design

allows any-to-any storage-to-host connectivity without having to
physically connect any-to-any storage-to-host.

9. The core/director switch forms the backbone of the Fibre

Channel SAN. It serves as the high density multi-speed routing
agent and signal regenerator for routers, edge switches or end
devices (e.g. servers, storage disks).

10. The edge switch provides a tiered level connection between

groups of host devices and the SAN.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 8 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

11. The following table provides the design parameters for the types
of switches to be used for the SAN:

Table 1: SAN fabric switches

Type Minimum Transmission Transceiver

Number of Ports Rate

Core/Director 128 scalable to 10 Gbps XFP and SFP

256 Fiber

Edge 32 4 Gbps with 1/2/4 SFP Fiber

auto-negotiate

B. Router

1. Routers shall provide scalability, security, and manageability by

enabling devices in separate SAN fabrics to communicate
without merging fabrics into a single, large SAN fabric.

2. The router shall function as a gateway for the other MARFIQ

locations and shall support the following SAN or LAN protocols:
FCP, FCIP, iFCP, iSCSI, IP.

C. Host Bus Adapter

1. Dual connected redundant Host Bus Adapters (HBA) shall

connect the motherboard bus of the host or storage device to
the SAN. The function of the HBA is to convert the parallel
electrical signals from the bus into a serial signal that is passed
to the SAN.

2. Common type of HBA’s shall be used for workstations, servers

or storage disk devices to ensure consistency.

3. Maximum distance from the HBA to switch is 300m.

4. HBA’s may be attached to the PCI/PCI-X bus for transfer rates

up to 200 MBps. To ensure full utilization of the HBA, each
device bus must be checked to ensure equivalent or higher
speeds.

5. HBA’s shall provide parallel connection with 32 or 64 bit bus

width, include fault diagnostics, load balancing and auto-
negotiation features.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 9 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

6. The HBA shall communicate with the SAN using Fibre Channel
protocols at speeds up to 4 Gbps.

7. Each HBA shall be equipped with dual full duplex fiber optic
connectors.

D. Transceiver

1. Transceivers are used to convert the internal communication

transport mechanisms of SAN switches and devices to a
common serial multi-gigabit transport protocol.

2. The following table lists the acceptable transceivers that may

be used for SAN devices:

Table 2: Transceivers types for SAN devices

Transmission
Type Rate Interface Features

Small Form- 4Gbps Optical, S-M Hot pluggable, full

Factor or M-M duplex up to
Pluggable 500m, short wave
(XFP)

10 Gig Small 10Gbps Optical, S-M Hot pluggable, full

Form-Factor or M-M duplex up to
Pluggable 100km, long wave
(XFP)

E. Cabling

1. SAN cabling shall be fiber using either short (780 nm multi-

mode) or long (1300nm single-mode) wave optical transmitters.
The long wave transmitters shall be compatible with either long
or short wave signals.

2. There shall be a consistent use of connectors for all SAN

devices. Mixing of fiber connector types is discouraged.

3. Fiber optic cabling shall meet the requirements of project

specification. The fiber cables selected for the SAN shall be
consistently used for all links. Mixing of fiber core diameters is
discouraged.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 10 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

4. The Structured Cabling System for the SAN shall follow the
requirements of project specification.

F. Encoding/Decoding

To optimize the signals for transmission across the SAN, each 8

bit byte of data originating at the host bus is transformed to a 10
bit symbol using 8b/10b encoding and decoding mechanisms.

4.2.3 Logical System

A. Port Types

1. The following FC port types may be configured for switches,

hosts or storage devices:

Table 3: Port type configurations

Port Descriptive
Name Name Device Type Port Function

N_Port Network Port Node Node port used to

(server/storage) connect a node to a
Fibre Channel switch

F_Port Fabric Port Switch Switch port used to

connect the Fibre
Channel fabric to a
node

E_Port Extender Port Switch Used to cascade

Fibre Channel
switches together

G_Port General Port Switch General purpose port

which can be
configured to emulate
other port types

B. Addressing

1. Device Identification: SAN devices shall be identified by the

WWN, a unique identifier which is defined and maintained by
the IEEE.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 11 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

2. Alternate Device Identification: Static or dynamic 24-bit port

addressing scheme may be used in lieu of the 64-bit WWN
address to increase routing performance. In such
circumstances, the switch shall be responsible for assigning
and maintaining the port addresses.

C. Framing

1. Framing formats: Frames, with a data field of 2112 bytes are

the smallest unit of information transfer. A sequence has at
least one frame and an exchange has at least one sequence.

2. Handling of other SAN protocols: SCSI, FCIP or other storage

network protocols shall be mapped and encapsulated inside
Fibre channel frames, sequences and exchanges at the FC4
protocol layer stack. All congestion, management, data error
and data loss recovery is handled by Fibre channel protocols.

D. Flow Control

1. Flow control protocol to be used: Fabric Shortest Path First

(FSPF) is a routing protocol defined by ANSI FC-SW-2 and
implemented for the SAN.

2. FSPF path selection: FSPF keeps track of the links on all

switches in the fabric and associates a cost with each link. For
multiple or redundant paths, FSPF determines the least cost
path for a frame to travel by computing and choosing the path
with the lowest number of hops. Should all hops have the
same latency, operating speed and not congested, FSPF will
compute how the frames are delivered to their destination by
selecting the fastest route.

3. Buffering capability: Ports need memory, or buffers, to

temporarily store Fibre Channel frames as they arrive and until
they are assembled in sequence, and delivered to the upper
layer protocol. Buffer credits control the number of frames sent
by the SAN device across the WAN since speeds may vary.
The credits are sent from the receiving end and administered
by the SAN device.

E. Zoning

1. Zoning allows for segmentation of the switched fabric. Only

the members of the same zone can communicate within that
zone, and all other attempts from outside are rejected.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 12 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

2. Examples when zoning may be required:

 To segregate different operating platforms since

each platform stores data in a different manner.

 Security purposes

 For testing and maintenance purposes

3. Zoning may be implemented using the switch to sectionalize

physical ports on the fabric (e.g. server A accesses storage
device A, server B accesses storage device B, etc.). This sort
of zoning allows for the highest level of security.

4. Zoning may be implemented using software in the switch to

sectionalize by WWN addresses. Since the WWN does not
change, this sort of zoning allows ports on a device to be
moved without effecting the overall configuration.

4.2.4 Disk Storage System

A. The SAN shall be able to operate any of the following storage device
types: solid state, disks, disk arrays, tapes and tape libraries.

B. The following table lists the network services and capacity required for
each MARAFIQ SAN cluster:

Table 4: MARAFIQ SAN Enterprise Servers & Capacity Requirements

Operating Expected Expandable

System Platform Server Application Storage (TB) to (TB)

TOTAL
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 13 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

C. The following table lists the network services and capacity required for
MARAFIQ other locations mirrored clusters:

Table 5: SAN Cluster Capacity Requirements for Other MARAFIQ Locations

Operating Expected Expandable

System Platform Server Application Storage (TB) to (TB)

TOTAL

D. The preferred storage device type is disk arrays. Disk array storage shall
be multi-drive, multi-platform with dual RAID controllers and dual
redundant hot swap controllers, cooling and power supplies.

E. The controllers and disks shall be housed in an external multi-bay

enclosure. Memory disk modules may be inserted to a common
backplane.

4.2.5 Performance Requirements

A. The availability of the SAN shall be greater than or equal to 99.999% or

approximately five minutes or less of down time per year.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 14 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

B. Latency is the time that it takes for a Fibre Channel frame to traverse the
fabric. Combined latency includes the latency in the ports, switches,
long-haul links, controllers, and inter-switch links. Combined latency for
the SAN shall be kept at less than 150ms for a single way frame
transmission.

C. The BER shall be 1X 10 12 .

D. Connections may be configured for Class 3 service: this service makes

efficient use of bandwidth; no dedicated connection established; frames
are not acknowledged leaving error recovery to upper layer protocols.

E. Connections may be configured for Class 4 service: this service is

connection oriented; frames received in sequence; fraction of available
bandwidth of a path is dedicated for the establishment of Virtual Circuits
which guarantee Quality of Service.

F. Bandwidth shall be allocated for Class F service: Class F is

connectionless service used by switch ISL communication for control,
coordination, and configuration of the fabric. Allocate 10% bandwidth to
each ISL for use by Class F service.

G. Traffic analysis shall be performed under varying fault conditions to verify

channel traffic overloads or congestion in the network does not occur.

H. A link budget analysis shall be performed between each node to

determine transmit power levels, system losses, and verification of
receiver sensitivity limitations.

I. Oversubscription: Each server throughput shall be verified against

storage throughput to ensure a 1:1 oversubscription ratio. For example,
four servers with throughput of 200MBbps accessing a single storage
device with 200MBbps results in a 1:1 oversubscription ratio.

4.3 Network Management

4.3.1 The SAN management system permits configuration, provisioning, LUN

assignment, zoning and masking, as well as monitoring and optimizing
performance, capacity, and availability for storage(e.g. disks, tape drives,
etc.) and network level (e.g. switches, directors, HBA’s) devices over a
remote link.

4.3.2 SAN devices shall communicate with the management system using either
SNMP or HTTP web based application layer protocols across the Fibre
Channel network.

4.3.3 Installation of software applications to SAN devices shall be performed

seamlessly causing no interruption to the network.
Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 15 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

4.3.4 SAN management tools shall provide the following problem troubleshooting
features:

1. Fabric interrogation

2. Locate missing devices

3. Identify failures in the cabling system

4. Interpret message and error logs

5. Suggest where to start troubleshooting

6. Record information in an orderly fashion

7. Create SAN diagrams

8. Access port information and configuration

9. Understand and interpret switch LED’s

10. Use vendor specific diagnostic tools

11. Locate error messages on the host or storage devices

12. Check for zoning conflicts

13. Use FC ping and trace routes

4.3.5 The SAN management system shall maintain a database with information
about connections, naming conventions, device serial numbers, WWN,
operating platforms, zoning and system applications.

4.4 Security

The SAN system shall be provided with a state-of-the-art comprehensive security

framework that will help ensure safe and reliable data processing throughout the
system. It shall address a wide variety of potential security threats such as snooping,
spoofing, denial of service, enabling a robust SAN infrastructure. The security
framework shall address authorization, authentication, data confidentiality, and data
integrity adequately. The SAN security technology will include but will not be limited
to:

4.4.1 Access Level Security

4.4.2 Encryption

4.4.3 Logical Unit Number (LUN) Masking

Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 16 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

4.4.4 Port zoning

4.4.5 World Wide Name (WWN) zoning

4.4.6 FCAP - Fiber Channel Authentication Protocol

4.4.7 FCPAP - Fiber Channel Password Authentication Protocol

4.4.8 SLAP - Switch Link Authentication Protocol

4.4.9 FC-SP, Fiber Channel Security Protocol

4.4.10 ESP - Encapsulating Security Payload

4.4.11 DH-CHAP, Diffie Hellman – Challenge Handshake Authentication Protocol

4.4.12 IP Security
 Power and Water Utility Company for Jubail and Yanbu (MARAFIQ) Master Specification: MQ-SP-T-7812
Project Name: Date: 04 Apr 2011
Contract Number: Page: 17 of 17
Revision: 0

SPECIFICATION – STORAGE AREA NETWORK

5.0 SYSTEM ARCHITECTURE DIAGRAM

The following simplified architecture diagram is given as a general topology overview to aid in
understanding the SAN concept. The exact components and configuration will be determined
during the detailed design phase.

This sketch is only an example and must be revised during detailed design phase.

Other MARAFIQ MAIN MARAFIQ LOCATIOM Other MARAFIQ

Locations Locations
(to be determined later)
(to be determined later) (to be determined later)

MARAFIQ Other MARAFIQ Location

MIRRORED CLUSTER MIRRORED CLUSTER MARAFIQ
At another location (Typ.) STORAGE CLUSTER

Via Local SAN 4 Gps FC

To another MARAFIQ
10 Gps FC Location
10 Gps FC ISL Trunk
Director Director

Multi-Protocol Router 4 Gps FC

At another MARAFIQ
Location ISL Trunk
EDGE SWITCH EDGE SWITCH

4 Gps FC

10/100/1000 LAN
Network Management ETHERNET
HOSTS