WHITE PAPER

Extending SAN Connectivity for

Engenio Arrays
SANbox 6142 Intelligent Storage Router Simplifying Fast SAN over WAN Connectivity

Executive Summary
The storage industry provides an abundance of white papers
that discuss the ever-changing data management and retention
regulations, the need for business continuity, and disaster
recovery. They propose solutions that meet these and other
business and regulatory needs, including:
Business continuity data replication for local and disaster
recovery (DR) protection
Flexible IT operations information distribution & application
development
Bridging multi-vendor SANs due to mergers & acquisitions
Improving application availability globally
Growing regional regulations related to data protection and
retention
The core of these data services is the ability to copy, move, and
replicate data, both locally and far away. Unfortunately, SANs only
reach about 10km. IT managers need to share, protect, and move
data much farther than that. Routers and switches linking SANs
over WANs are the technology of the moment to help IT managers
protect valuable data over long distances to support business
continuity with migration and replication, to consolidate islands
of SANs, and to improve the usage of WAN and SAN resources.

Engenio storage arrays and RAID controllers are used in a

wide range of storage topologies. The SANbox 6142 Intelligent
Storage Router can bridge SAN topologies over the WAN and it
is fully interoperable with Engenio Series 2882 and 5884 RAID
systems, the 6091 external RAID controller, and LSI Volume Copy
software.
Key Findings
The right SAN-over-WAN solution optimizes enterprise resources
with lower cost and management overhead. The SANbox 6142
router makes SAN-over WAN-connectivity fast and simple.
SmartWrite compression and load balancing to optimize
SAN-over-WAN performance:
Simplifies configuration and management
Compresses data, reducing WAN latency/increasing line speed
Delivers twice the performance and failover between IP
links
SANbox 6142 offers a low-cost, high-performance solution for
distance migration and replication on Engenio storage arrays:
Sets new price-point for SAN-over-WAN distance connectivity
Leverages SCSI Layer 3 routing
Eliminates multiple round-trip delays

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

SAN over WAN Examining the Options

The key challenges in building SAN-over-WAN solutions are matching
performance and management requirements to cost, network model,
and transfer protocol choices that meet the needs of each business
application. When reviewing how and why to move SAN data over the
WAN, we must examine the SAN-over-WAN solution stack, shown in
the following illustration.
Regulatory
Business Continuity
Asset Maximization
Replication and D2D Backup
Disaster Recovery
SAN-Island Connection
FCIP and WAN Optimization
WAN Latency
Acceleration Protocols
Routers
Switches/Blades
Applications
Distance Requirements
Performance, Transfer Size
Costs

Business Drivers for the SAN-over-WAN Market The

5 Cs: Compliance, Control, Continuity, Consolidation
and Cost
The five Cs are critical to understanding and analyzing the business
needs for SAN-over-WAN connectivity. This includes:
Control Managing and securing data concerns IT managers
since any mistakes can show up on the front page of the
Wall Street Journal or Financial Times. Transferring data over
telecommunications lines, especially public lines, requires
stringent security and control measures at every level: data
delivery, user access, and authentication.
Continuity Ensuring business continuity is a leading priority
for all IT organizations. From both regulatory and practical
standpoints, the source and backup copy of data need to be
geographically separated to protect against regional issues such
as power grid failures, natural disasters, and so forth. Linking
SANs over WANs is the primary means to accomplish this type
of data replication.
Cost As the demand for SAN-over-WAN services for data
protection grows, so does the need to control the cost of WAN
lines. Key solutions to lower the cost of SAN-over- WAN solutions
include: tools to improve WAN line efficiency, data compression,
and load balancing to minimize the cost/Mb transferred.
Consolidation Traditionally, storage consolidation refers
to aggregating storage within the data center. In this case,
consolidation includes consolidating SAN islands from multiple

locations to simplify replication across multiple SANs and storage

systems.
Compliance Worldwide, government regulatory requirements
have changed to put greater demands on businesses. The data
outlined by these regulations is often outside the data center,
but still needs to be protected, secured, and accessed globally.
US regulations such as SEC 17a-4, Sarbanes Oxley section
404, and HIPAA drive the need for new storage strategies in
every organization. This means that those servers, and the
storage that supports those servers, need data protection and
management capabilities, often associated with enterprise SANs
and traditionally found only in central IT departments.

SAN-over-WAN Applications and Uses

The applications for SAN-over-WAN transfers fall into three major
groups:
Data replication for backup and business continuity
Disaster recovery planning and hot-site failover
SAN data and device sharing
These applications and their uses are as diverse as the businesses
they support. However, in each case, the underlying need is to
connect SANs over distances greater than the inherent 10km limit
of FC SANs.

SAN-over-WAN Applications Assessment

The next step is to assess the requirements of the SAN-overWAN application. To help define the requirements of your specific
application, consider the following:
Data Targets How much data do you have? The volume of the
target data is fundamental to establishing the WAN connections
performance requirements.
Availability Can the connection be a best effort or is 24/7
availability a requirement? This is a key to determining the type of
software, hardware, and WAN connection, as well as the overall
cost to build and run a SAN-over-WAN connection.
Write Transfer Objective How much time is available to
transfer the data? This determines the size and number of
telecom lines needed.
Recovery Time Objectives (RTO) How fast must the data be
recovered? How often will you be required to recover it?
Recovery Point Objective (RPO) What point-in-time do you
want to recover the data? Are you planning to link systems
synchronously or asynchronously or use new solutions like CDP
(Continuous Data Protection)?

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Budget What is the TCO compared to the ROI? You must

consider extra disk storage, required software, and particularly
ongoing WAN line cost, as well as equipment cost to get a full
TCO/ROI analysis.
Data Access Model What kind of data access do you require:
asynchronous or synchronous transfers? Asynchronous transfers
can lower cost and performance requirements compared to
accommodating peak synchronous demands.
By answering these questions, you can systematically exclude or
include specific solutions. For example, if the SAN-over-WAN transfers
include different classes of data, you may want Quality of Service
(QoS) functionality. Knowing these answers and your application
availability requirements, your IT team will be in the best possible
position to narrow the choices among solutions.

Understanding WAN Performance

The key factors in understanding WAN performance are:
WAN Latency (a function of distance)
Network Latency at each end of the WAN
WAN pipe performance
The number of WAN pipes used (load sharing)
The following figure shows a typical WAN configuration for replication
or SAN bridging. Most typical WAN configurations require a two-way
communication process that introduces latency into the transfer
process.
When examining the complete transfer time required for a given
volume of data, the first place to start is the overhead of the WAN and
the network infrastructure.

1. The initiating system issues a write request to the remote system: 50ms delay
2. The remote system responds with a Ready status: 50ms delay
3. The initiating system sends the first block of 32KB data: 50ms delay
4. The remote system issues a transfer ready status: 50ms delay
5. The initiating system sends the second block of 32KB data: 50ms delay
6. The requesting system confirms that the transfer is complete: 50ms delay

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

WAN Latency

Transfer Data

The primary cause of WAN latency is the distance that the light
beam, used as a signal, has to travel between the source and the
destination locations. The rule is that 100 miles of distance has a
1ms (millisecond) delay. So, a replication solution connected over
1000 miles has a 10ms one-way latency and a 20ms Round Trip
Time (RTT). Since, in most WAN communications, one device sends a
command/data transmission and the other device returns a response/
acknowledgement, RTT is the most common method of quoting WAN
latencies. The following chart shows the expected one-way latencies
at specific distances.

Size of File

64 MB

Size of Each Transfer

64 KB

Latency Local (both ends)

20ms

WAN Distance Latency

70ms

Number of Transfers

1024

Number of Round Trips

2048

Total Latency time in Seconds

184

You can see that the 64MB file, broken into 64KB transfers, requires
1,024 write operations. Assuming a WAN latency of 70ms and a
network infrastructure latency of 10ms at each end, you have a 90ms
total for each write instruction and for each of the acknowledgements.
The total latency time is 184 seconds, or three minutes to transfer the
file regardless of the speed of the WAN line itself. The second primary
factor in WAN performance is the line speed of the WAN connection.

One example of the impact of WAN latencies on transfers is a

replication transfer from New York to London. The distance is about
3,456 miles, causing about a 35ms one-way delay or 70ms round
trip time. The following chart outlines a transfer of a 64MB file over
this distance with standard IP over WAN transfers (two round trips
per write instruction).
 See Appendix A.

IP over WAN

Latency Over Distances

Round Trip Latency

250

Latency in Milliseconds

200

150

100

50

KM
09
3

KM
16
,

0
10
,0
0

M
0
00
9,

ile
s

ile
s

12

,8

14
,4
84

75

KM

KM
ile
s
M
0
00
8,

0
00
7,

0
00
6,

ile

ile
s

ile
s

8,

9,
65

04

11
,2
65

KM

KM
7

KM
M
5,

00

M
0
00
4,

3,

00

ile
s

ile
s

6,

4,
82
8

43
7

KM

KM
9
21
3,
s
ile

2,

00

M
0
1,

00

10
0

ile
s

ile

1,
60
9

16
1

KM

KM

Round Trip Latency

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

WAN Line Speed

Like all high-performance-oriented products and services, WAN
lines vary widely in cost and performance from a modest 1.5 Mb/s
to over 10Gb/s and the cost variance is almost as significant as the
performance options, from under $500 per month to over $250,000
per month. The following chart shows some typical WAN line speeds
and costs.
Line speed also has a material effect on the time it takes to transfer
files over the WAN. The following chart shows what the transfer time
variance is for the same 64MB file over the New York to London WAN
discussed earlier. As you can see, the baseline shows the latency of
the WAN distance and the bars show the total transfer time in minutes
for the 64MB file as you use higher performance WAN lines.

WAN Lines

Mbps

MB/S

Price/Month

Cable

3.0

0.4

$50

T1

1.5

0.2

$500

T3

45.0

5.6

$4,000

OC 3

155.0

19.4

$20,000

OC 12

622.0

77.8

$75,000

OC 48

2,400.0

300.0

$150,000

OC 192

10,000.0

1,250.0

$250,000

250.00

Transfer Time in Minutes

200.00

150.00

100.00

50.00

0.00

WAN Latency

T3

OC3

OC12

OC 48

OC192

Total Time in minutes

5.33

194.96

60.39

19.05

8.89

6.19

Latency Time in Minutes

5.3

5.3

5.3

5.3

5.3

5.3

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Multiple WAN Lines

The next issue is the number of WAN lines used to transfer the file. A
second line can reduce the total transfer time by almost 50% (some
overhead is involved). So the number of WAN lines is the last major
factor in determining transfer time and cost. Often, businesses meet
their needs by using multiple low-speed lines rather than moving up
to the next, more costly, service level.

Options for SAN-over-WAN Connectivity Models

Based on IP networks, there are two hardware models that connect
SANs over WANs: using a router blade in a core switch or using a
router/gateway. In both cases, the hardware provides the physical
link between the SAN (usually Fibre Channel) and the WAN based on
IP running on Ethernet. The choice of switches versus routers and

gateways is usually a question of matching your network architecture,

budget, and application requirements.

Extending the SAN with FCIP

Fiber Channel over IP (FCIP) creates a tunnel that transports the FC
commands and traffic from one FC SAN to another over IP using an
FCIP gateway/router/switch installed on each SAN. Each FCIP tunnel
is a virtual connection point, and each tunnel has an IP address.
This process creates a link directly between the SANs and the data
transferred. The following figure shows the operation of the FCIP
protocol.
For more information on FCIP, see the FCIP standard at:
http://www3.ietf.org/proceedings/02jul/I-D/draft-ietf-ipsfcovertcpip-11.txt

SAN Over WAN Connection Methods

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

SmartWrite Advanced Layer 3 Routing

SmartWrite:

SmartWrite, unlike FCIP, is a patent-pending intelligent routing

technology that understands, not only how to move the data from
one point to another, but how to optimize that transfer. It is the only
intelligent SAN-over-WAN protocol to leverage SCSI Layer 3 routing.
This allows SmartWrite to not only optimize transfers over the WAN, it
simplifies configuration and management.

Eliminates double addressing of SAN devices (IP and FC)

Eliminates the need to have unique names on each SAN
Eliminates 50% of the management overhead of FCIP
Leverages the WAN resources for resiliency and encryption
In addition to lowering management overhead, SmartWrite optimizes
performance over the WAN.

1. A SAN application issues a command to send an FC packet to an FC target over SmartWrite Layer 3 SCSI routing.
2. SmartWrite intelligently:
a. Determines the total wire transfers to reduce WAN latency
b. Compresses the data to improve WAN line speed
c. Leverages load balancing to minimize transfer time over multiple lines (if available)
3. SmartWrite converts the FC packet to an iSCSI packet and routes the data without merging the SAN fabrics.
4. SmartWrite sends multiple transfers in one process, reducing WAN latency.
5. SmartWrite delivers the FC packets to the target device.

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

First, SmartWrite can detect the total transfer size to go over the
WAN. This enables SmartWrite to tell the receiving router how many
transfers to expect over the WAN. For example, a source router
(located in New York) can tell the destination (located in London) that
it will receive a write, as well as how many transfers it will receive to
complete the file. This enables SmartWrite to send data in a constant
stream from the source to the destination without having to wait
for individual acknowledgements for each write. This dramatically
reduces the latency of the transfer, increasing line usage for data
transfer, which can reduce WAN line costs.
Second, SmartWrite can compress the data before transferring
it, which reduces the volume of transferred data. This reduces the
number of transfers required to move the data, reducing the transfer
time and lowering WAN costs.

SANbox 6142

Third, SmartWrite can load balance transfers over multiple WAN/LAN

lines to improve performance, lower total transfer time, and reduce
the number of transfers over the WAN.
All of these factors allow for a cost-efficient, high-speed, and easyto-manage, core-to-core bridging strategy. The following illustration
shows how SmartWrites performance optimization works in the
same New York to London WAN example.
In this example, the total transfer time has been reduced to 100ms.
As mentioned previously, real-world protocol handshaking requires
hundreds of transfers, each with its own latency, compounding the
total transfer time. By combining most of these commands locally
and then sending, SmartWrite enables remote core switches, and the
associated SAN island, to be easily bridged into the other resources.

SANbox 6142

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Picking the Right IP SAN Extension Option

Network architecture is one of the key criteria when deciding
between using a blade in a switch or a stand-alone router/gateway.
The obvious choice would be to say, If I have directors or switches
that can use a blade for SAN-over-WAN connection, I should leverage
that investment.
However, that might not be the best use of the precious director or
switch blade slots. The primary goal of most large switching devices is
to move data rapidly, at maximum throughput and to remain available
at all times.
When IT managers have spent more than $100K to 500K on
director class hardware or HA switches, typically, the best use
of that bandwidth should focus on providing low latency transfers
through the core of the data center that supports mainframe traffic
to the open enterprise or other premium network services. When IT
managers examine the implementation and/or the opportunity cost of
the director class SAN-over-WAN blades, stand-alone routers become
a viable and attractive alternative. The opportunity to use routers not
only becomes a viable choice, it provides a strategic advantage for
your network architecture.
The following table shows director opportunity costs, vital numbers for
deciding on the router/gateway or the director/switch blade solution.

This table shows three vital numbers for making the router/gateway
versus director/switch blade decision:
Two-Site Cost of Routing To create a high-availability
configuration, two routers need to be placed within each location
(four total). This greatly impacts the capital investment of directors
and switches. The amortized cost of the embedded routing solution
sacrifices some very dear resources that could be better used for
their primary switching functions. In many cases, the capital cost
of the stand-alone router model represents a 60% saving over the
HA switch model and over an 80% cost savings compared to the
director class model.
Opportunity Cost Factor The opportunity cost of using one
of the precious blade slots on a director or switch adds to the
capital cost of the equipment. Each of those slots represents an
opportunity cost of over $20,000. The router model for linking
SANs over WANs does not require this type of opportunity tradeoff.
Price per IP Routing Port Since the number of IP ports and line
latency are the primary drivers of performance, the other key cost
metric is the price per IP routing port. In each of these products,
the base model router provides two ports. Two routers provide four
router ports on each end for a stronger ROI and lower capital cost
with no opportunity cost versus the HA switch or director model.

SANbox 6142

Brocade 7500*

Cisco 9216i*

Cisco 9506*

$12,999

$42,322

$55,000

$134,000

Switch/Director Slots for Blades

--

$18,333

$26,800

Opportunity Cost/Slot

--

$23,333

$35,200

Base Unit

Price Per IP Routing Port

$6,500

$21,161

$27,500

$67,000

Dual router Boxes for HA

$25,998

$84,644

--

--

Chassis Costs

--

$55,000

$134,000

Blade Costs

--

--

$20,000

S F Ps

--

--

$3,500

SAN Routing Software Costs

$15,350

$15,000

$15,000

Management Software

$2,700

$3,500

$3,500

$102,694

$165,166

$313,300

Two-site Routing Cost w/ HA

$51,996

*Based on manufacturer list prices

SSG-WP07011 
SN0130970-00 A

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

QLogic SANbox 6142 Intelligent Storage Router

The SANbox 6142 integrates well into any network topology.
The gateway provides a bridge between remote SAN fabrics,
while maintaining SAN independence. The most straight-forward
implementation is having a gateway at each site to connect the
respective fabrics across the LAN or WAN.

to the LAN/WAN. To achieve this, a second SANbox 6142 is installed in

each location, as shown in the following illustration.

SANbox 6142

SANbox 6142

The illustration shown above shows quick and cost-effective

deployment of disaster recovery and remote replication solutions.
Even in data centers with a split-fabric architecture, the Intelligent
Storage Router can support multiple paths for the various fabrics, as
shown in the following illustration. The A and B fabrics in each
location can have independent paths to the SANbox 6142.

For the most stringent of high-availability requirements, a secondary

WAN path should be routed. This can minimize or eliminate most of
the disruptive events outside the IT Departments control. The ports,
routing, and configuration of the SANbox 6142 solution allows for
this type of deployment. Better still, a company can start with the
minimum configuration and then seamlessly grow to maximum faulttolerant system, as shown in the following illustration.

SANbox 6142

SANbox 6142

However, in many cases, mission-critical functions require a fullyredundant, high-availability architecture. This is typically the case
when the business has very small recovery time objectives (RTO) or
very recent recovery point objective (RPO). The mantra of no single
point of failure holds true in this situation. To guarantee a successful
RPO/RTO scenario, a dual path must be implemented from the fabrics

SSG-WP07011

The SANbox 6142 bridges Fibre Channel SANs to provide IT

administrators with a cost-efficient method of SAN extension by
connecting SANs over WANs. The SANbox 6142 provides SAN-overWAN connection for migration, tape backup, and SAN bridging. The
SANbox 6142 sets a new price/performance mark in the industry for
SAN-over-WAN connection. Key advantages of the SANbox 6142 are:
Lower WAN costs with SmartWrite
Improved ROI by sharing SAN resources globally
Multi-Protocol support for SAN devices lowers cost

SN0130970-00 A

10

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Leading applications for the SANbox 6142 include:

Implementing remote data migration, replication, and disaster
recovery

Engenio StoreAge Software

Connecting SAN islands over WANs and LANs

Bridging heterogeneous SAN fabrics

Engenio Replication Software

Engenio offers two families of replication and data migration
software. The first is an embedded set of tools that run on their array
controllers. Volume Copy is used for moving complete SAN volumes
or snapshots, and Remote Volume Mirror for performing synchronous
and asynchronous replication over long distances between arrays. The
SANbox 6142 is certified by Engenio to support these applications.
Volume Copy
Volume Copy, a premium feature of SANtricity and Simplicity
software, creates an exact duplicate, or clone, of the production data
which can be mapped to a separate server for analysis. The clone can
be used for application testing or development, information analysis,
data mining, or backup/restoration.
Volume Copy can also be used to redistribute data (moving volumes
from older, slower disk drives to newer, faster or higher capacity
drives). This optimizes performance, storage density, and capacity
utilization.
Remote Volume Mirror
Remote Volume Mirror (RVM), an optional feature of SANtricity
Storage Manager software, protects the information stored on your
LSI Engenio storage system through real-time data replication to an
off-site system.

SSG-WP07011

For each set of volumes that comprise a mirror pair, RVM supports
a variety of replication options to provide you with the flexibility to
optimize data protection and utilization.

The second option for replicating and migrating data on Engenio

arrays is to use their StoreAge virtualization software. The SANbox
6142 is also certified to provide SAN extension capabilities for the
StoreAge software.
StoreAge multiCopy
StoreAge multiCopy is an innovative copy facility that enables the
creation of multiple physical copies of volumes, regardless of storage
subsystems and SAN components. Each copy is independently
accessible and instantly available. This allows production servers
to access data while the copy operation is in process. StoreAge
multiCopy enables copying and re-purposing information, while
leaving production servers available for productive work and
uninvolved in data movement.
StoreAge multiMigrate
StoreAge multiMigrate enables the online migration of data from any
storage device to any other storage device regardless of vendor. The
migration takes place while the applications remain on-line without
any interruption. This application is excellent for migrating critical
applications from older storage devices onto newer platforms.
StoreAge multiMirror
StoreAge multiMirror is an innovative mirroring facility implemented
in conjunction with LSI SVM. StoreAge multiMirror offers enterprises
an advanced disaster recovery solution using synchronous and
asynchronous local and remote mirroring. StoreAge multiMirror
provides simple and fast data recovery utilities and enables a uniform
storage service, regardless of storage subsystems brand or quantity
and independent of network bandwidth.

SN0130970-00 A

11

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Summary and Conclusion

High Performance Routing using SmartWrite

This paper reviewed the state of SAN-over-WAN solutions and how

to plan and model a SAN over WAN implementation for Engenio
systems. IT organizations have made significant investments in their
SANs. To protect those investments and incorporate new solutions, IT
organizations can benefit from the SANbox 6142 features:
Layer 3 routing scales to meet enterprise needs
Keeps SAN independent
Improves fault tolerance
N_Ports Bridge multiple vendors SANs
Bridge SAN without losing vendor specific features
Share SAN devices like they are locally attached

Reduces WAN latency by 60% versus FCIP

Compresses data
Performs load balancing over IP ports
As with many applications and solutions discussed in the storage
industry, price, performance, and protection are driving factors. The
QLogic SANbox 6142 sets new standards for:
Improving SAN over WAN performance
Reducing the number and cost of WAN lines required
Protecting data with resilient WAN connections
Setting new price points for SAN over WAN solutions
Lowering management
implementations

overhead

for

SAN

over

WAN

For more information on these solutions please contact us at

http://www.qlogic.com.

SSG-WP07011

SN0130970-00 A

12

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Appendix A Distances between Major Cities

Distances in miles; source: http://www.mapsofworld.com/utilities/world-airdistance-locator.htm

Mu

SSG-WP07011

SN0130970-00 A

13

WHITE PAPER

Extending SAN Connectivity for Engenio Arrays

Disclaimer
Reasonable efforts have been made to ensure the validity and accuracy of these comparative performance tests. QLogic Corporation is not
liable for any error in this published white paper or the results thereof. Variation in results may be a result of change in configuration or in
the environment. QLogic specifically disclaims any warranty, expressed or implied, relating to the test results and their accuracy, analysis,
completeness or quality. All brand and product names are trademarks or registered trademarks of their respective companies.

SSG-WP07011

SN0130970-00 A

14