RecoverPoint With SRM

This module covers prerequisites and configuration of EMC RecoverPoint.
Copyright 2012 EMC Corporation. All rights reserved
VMware vCenter SRM Integration with EMC SRAs
This lesson covers EMC RecoverPoint configuration, splitters, and consistency groups.
The VNX splitter is an array based splitter for RecoverPoint that runs on each VNX service
processors and splits all writes to a VNX volume that are intended for RP volumes. It sends
one copy to the original target and the other copy to the RecoverPoint appliance. You must
create fibre channel zones among all HBA adapters on RecoverPoint appliance to all available
VNX storage processor ports. Also while registering the RecoverPoint appliance with VNX,
you must select initiator type as RecoverPoint Appliance 31 and Failover mode to 4. Here are
some of the storage volumes must be created for the storage arrays for the RPA cluster:
1. Repository volume - This volume stores replication environment information such as
consistency group metadata and must be only be accessed by the RPA on the same site.
The minimum Repository volume size is 3GB.
2. Journal volume This volume stores the consistent point in time images for the target
site. The general recommendation on the size of the journal volume is 5GB for non
distributed consistency group and 20GB with the distributed consistency groups.
3. Replication volume This is the data volume that will be replicated. The source and
target volume should be the same size or the target volume should be greater than the
source volume.
vStorage API for Array Integration (VAAI) was introduced with vSphere 4.1 and later version
of VMware vSphere. VAAI offers certain VMware operations handled by the storage arrays to
improved the performance and usability of vSphere. The VAAI features are:
1. Full Copy This feature uses xcopy SCSI command to speed up the virtual machine
deployment.
2. Block Zero This is known as copy same and will speed up the bulk zeroing of a disk.
3. Hardware assisted locking This is also known as Atomic Test and Set and implemented
using compare and swap SCSI command. This feature implements an efficient LUN
locking in the ESXi clustered host environment.
4. UNMAP This feature allows ESXi host to inform the storage array that files or VMs have
been moved or deleted from a Thin Provisioned VMFS datastore. This feature is only
available from vSphere 5.0 U1.
RecoverPoint supports each VAAI command at one of the following levels:
Blocked when RP blocks or rejects the use of a VAAI command, vSphere automatically
reverts to the legacy behavior.
Supported RP supports the VAAI commands and its functionality.
Unsupported RP does not support a particular VAAI command and it must be disabled
at the ESXi host. Failure of disabling the unsupported VAAI feature can cause data loss.
You can disable the VAAI feature by setting the option 0 if they are not supported by the RP
using vSphere Client. Select the ESXi host and go to the Configuration tab, and click
Advanced settings under software. VAAI options are located under DataMover and
VMFS3 section.
Create storage group on the protection site VNX array for the RecoverPoint appliance and
add repository volume (3GB), Journal volume (5GB), and host data volume in the storage
group. Ensure that the repository and journal volumes are not accessible by protection and
Recovery ESXi hosts. The repository volume holds configuration data that is created in the
installation and replication process. The minimum journal volume size is 5GB. and the actual
size of journal volume is dependent on the amount of snapshot data that is likely to be
written or a consistency group in 24 hours period.The replication volumes (source and target)
should be visible to the hosts and the RPAs. The slide shows the protection site VNX storage
group configuration for RPA. It contains production volume (ESXi-136-prod) with a Jvol (5GB)
and LUN 0 (3GB) repository volume. Where as the protection ESXi host has only access to the
production volume (ESXi-136-prod).
Almost similar storage groups are created on the recovery site as you had created on the
protection site. Here you can see two storage groups. One for the RPA and other is for the
Recovery ESXi host. The RPA storage group contains the target volume (replica of the
protection site production volume), one Journal volume Jovl, and one repository volume. The
storage group for recovery ESXi host contains only the target volume.
The RecoverPoint (RP) consistency group ensures that the replicas (replicating volumes) are
always consistent and in current write order. If your application requires two or more
datasets and they are dependent on each other then you should consider adding the
datasets into one consistency group to ensure the write-order fidelity. A RP consistency
group consists of one or more replication sets. Each replication set consists of a production
volume and a replication volume. As you can see on the screen that the RecoverPoint
configuration contains one consistency group SRM-50 that has only one replication set
RSet1. The replication set RSet1 has one production volume ESXi-136-pord(4) and one
replication volume ESXi-137-DR(4).
The RP consistency group allows you to set the replication policies such as compression,
bandwidth limits etc.
Once the RP consistency group is configured correctly and the EMC RecoverPoint SRA has
been installed, you should set the option Group is managed by SRM, RecoverPoint can only
monitor under RecoverPoint CG policy option.
It is a good practice to make sure that replicas (replicated volumes) can be used to restore
data from a disaster situation. In most cases, while testing a replica, applications can
continue to run on the protected site and replication can continue as usual. The writes will
be stored in the replica journal until testing is completed. When testing is completed, the
write access at the replica is disabled and the writes will be distributed from the journal to
the replica. The entire process can be completed without application downtime and without
loss of data at the replica.
This lesson covered EMC RecoverPoint configuration, splitters, and consistency groups.
10
This lesson covers EMC RecoverPoint SRA prerequisites, installation, and configuration.
11
EMC RecoverPoint SRA for VMware vCenter SRM allows you to implement a disaster
recovery solutions for vSphere environment that uses EMC RecoverPoint appliances for data
replications. RecoverPoint SRA requires certain prerequisites which you will see on the next
slide for its proper functioning.
12
There are RecoverPoint SRA prerequisites you need to be aware about while installing and
configuring it. RecoverPoint appliances must be installed and configured for data replications
at SRM protected and recovery sites. You ensure that the RecoverPoint appliances have code
3.4 SP1 or later version running on it for RecoverPoint SRA 2.0 version to work. RP SRA 2.0 is
enabled for both CLR and CRR consistency groups but VMware vCenter SRM only uses the
CRR remote copy not the CLR local copy. RP SRA 2.0 does not support Raw Device Mapping
virtual mode (RDM/V) . Only RDM datastore in physical mode is supported by the RP SRA 2.0.
You must ensure that the vCenter Server 5.0 and vCenter SRM are installed and configured at
both protected and recovery sites and vCenter extension plug-in is installed on the vSphere
Client you will use to manage the SRM setup. RP SRA 2.0 requires JRE version 6 update 7 or
later to be installed on the SRM host.
You need to set some of the RecoverPoint policies that will be used by the RecoverPoint SRA.
Enable Use VMware Site Recovery Manager (SRM) under Stretch Cluster/VMware SRM
Support /Replication Enabler for Microsoft Exchange Server 2010.
Enable Reservations Support under Advanced/General section.
13
The RecoverPoint SRM installation is performed on both protected and recovery vCenter
servers. The SRA installer InstallShield wizard does not require any configuration steps while
installation process. After RP SRA installation at both protected and recovery sites is
completed, you can add the RP SRA to the SRM configuration via Array manager
configuration using vSphere Client SRM window. You will be presented with the RecoverPoint
access information such as RP management IP address, Administrator user ID, and password
while adding the RP SRA to the SRM array manager . You ensure that the RecoverPoint
consistency groups used by SRM are enabled and in a replication state.
14
RecoverPoint SRA does not support the host based splitter. It only supports array based and
intelligent fabric based splitters. In this course we are using VNX array based splitters for
RecoverPoint configuration. After successfully installation of RP SRA at both protected and
recovery sites, you will enable the array pairs as shown on the screen. You can enable the
array pair from either site and only you have to enable once. After enabling the RP SRA you
will be able to see the local and remote devices for each enabled array pair. The Devices
screen will also show the direction of replication and datastore for the protected virtual
machine.
15
This lesson covered EMC RecoverPoint SRA prerequisites, installation, and configuration.
16
17
This lesson covers EMC RecoverPoint SRA configuration and SRM setup and testing.
18
Array managers configuration allows you to configure the storage replication adapters at
both protected and recovery sites. The configured SRA will allow the SRM to discover the
replicated devices, compute datastore groups, and execute storage operations command to
the connected storage array instructed by the SRM. You need to configure the array manager
once unless the array manager connection or access information have changed. You will also
need to configure the array managers in some situation when you want to use a different
arrays.
19
After adding the arrays using Array Managers, you should enable the array pairs. You can
enable the array pairs from either side of the SRM configuration.
20
Inventory mapping is an options steps but it is recommended. The inventory mapping is

created on the protected site and it establishes recovery site defaults for the virtual machine
folders, networks, and resource pools to which recovered virtual machines are assigned. The
inventory mappings apply to all virtual machines in all protection groups at the protected
site. To complete the inventory mappings correctly you must ensure that a placeholder
datastore has been created at the recovery site. The screen shows the inventory mapping
objects for protected recovery sites such as cluster, network mappings, and placeholder
datastore.
21
After setting up inventories mapping and creating a placeholder datastore you are ready to
create a protection group at the SRM protected site. The protection group will contain the
virtual machines that you would like to protect as a group via SRM. All virtual machines in a
protection group store their files within the same datastore group and all virtual machines
failover together in case of protected site failure. In this example, protection group PG1
contains the virtual machine Windows-2008-VM1 and the virtual machine is located on the
datastore group SRMS-VM-SRM. The datastore group listed here is the one that was
discovered as replicated datastore from the configured storage array.
22
The recovery plans are created on the recovery site that control the way in which virtual
machines in protected group are recovered at the recovery site in case of disaster situation
at protected site. The recovery plan is stored in the SRM database at the recovery site and
executed by the SRM server at the recovery site. A simple recovery plan assigns all virtual
machines in a protection group with two networks a recovery network and a test network.
The test network can be created by the SRM on an ESXi host and this network in exclusively
used for testing the virtual machines. SRM also supports the recovery network that can span
across ESXi hosts using vNetwork Distributed Switch (vDS). You have to manually create the
vDS to be used by the testing and failover recovery purposes.
You can customize the recovery plans that suits the business requirements. A simple
recovery plan includes a predefined steps that use the default values to control the order in
which the protection group members are migrated or failover to the recovery site. Recovery
plan is customizable, you can add your own scripts or change the steps order or steps
execution.
23
It is important to customize the IP address of the virtual machines to maintain its

connectivity while testing, failover, and failback operations within SRM setup. The
customization you apply to the protected virtual machines are shared between all recovery
plans that apply to the virtual machine. If you remove the protection of a virtual machine, all
recovery customizations are lost. The screen shows a IP customization screen for a single
virtual machine. However, you can also apply IP customizations to multiple virtual machines.
SRM provides a command line tool dr-ip-customizer.exe for customizing large number of
windows virtual machines.
24
SRM provides command tool dr-ip-cutomizer.exe to access the vCenter server in your SRM
environment and specify IP address properties for any or all of the virtual machines in a
recovery plan by editing the file that dr-ip-customizer.exe tool generates. There is an example
give on the screen how to retrieve the file from the vCenter server. After you have updated
the file with the customization information for all the virtual machines you should replace
the keyword generate in the command line with apply for customizing the IP
information.
SRM automatically sets LVM.enableResinature to 1 during the test failover. Once the replica
of the VMFS is mounted on the recovery site, the VMware kernel automatically renames
VMFS volume that has been resignatured by adding snap- prefix to the original name to
identify that it is a VMFS copied file system. vCenter SRM provides an advanced setting
storageProvider.fixRecoveredDatastoreNames to automatically remove the snap-prefix
during the recovery plan. This option is disabled by default. SRM does not allow the VMFS
volume to be forcefully mounted with a invalid signatures without resignaturing it. ESXi host
allows forcefully mounting the VMFS volume with an invalid signature and without
resignaturing it.
25
SRM allows you to test the recovery plans by simulating a failover of the virtual machines
from the protected site to the recovery site. The testing of recovery plan does not hinder any
virtual machine protections status and normal operations.SRM leverages EMCs storage
snapshot or cloning capabilities to perform recovery tests without pausing the protected
virtual machine. After running the test, SRM produces a web based test results showing the
success and failure of the tests. While testing the recovery plan, the virtual machines are
started up in a isolated environment at the recovery site. The isolated environment includes
a network and storage infrastructure.
Ensure that the protected and recovery sites are connected and working properly before
executing the test functions. Testing a recovery plan runs all the steps outline in the plan with
an exception of powering down VMs at the protected site and forcing devices at the recovery
site to assume the ownership of replicated data.
26
Here you can see the recovery steps performed during testing.
27
This screen shows the state of the virtual machine after test is performed on the protected
virtual machine. You can see the Windows 2008-VM1 virtual machine state is up and
configured with the test network and snap datastore at the recovery site.
28
The simulated failover (test recovery plan) is completed by resetting the environment after
testing the VM at recovery site so that it is ready for the next event. During the cleanup
process, SRM shutdown the test virtual machine and unregister it at the recovery site and
then register the placeholders back.
29
If you experience any errors during the cleanup process, you have an option to select the
Force Cleanup option to ignore the errors and forcefully complete the cleanup operations. If
this step is used, you may have to cleanup the storage manually and run another test to
confirm that the error has been resolved.
30
After performing the cleanup operation, the virtual machine state at the recovery site shows
that VM is being replicated to the recovery sites. The state of the VM is represented with an
icon showing a lighting bolt. As you can see with the virtual machine Windows-2008-VM1 on
this screen. The associate of the VM is with the placeholder datastore and no network
association.
31
SRM automatically generates a web based report after every test or recovery operations. The
report contains a log of the steps executed during the recovery plans execution, total time it
took to execute each steps in the recovery plans. This report can be used for the audit trail
and compliance purposes.
32
33
34
This lesson covers SRM failover, re-protecting, and failback.
35
After testing the recovery plan, you can actually perform the failover operation by selecting
the Recovery option on the recovery plan. When you run the recovery plan, all virtual
machines in the recovery plan are moved to the ESXi host cluster at recovery site. While
configuring the recovery plan for the virtual machine, you have an option for customizing the
IP settings, setting up priority group, configuring VM dependencies, and applying pre and
post power on steps. Example of priority group settings are shown on the screen.
36
Here, the default recovery plan steps are listed that get executed during the recovery plans.
You have an ability to customize these steps and change the order or executions. In some
cases you can also include the scripts that need to be executed as a part of recovery plan.
37
You have an option while running a failover to choose either Planned migration or Disaster
Recovery. When you select the Planned migration, SRM will attempts to replicate all virtual
machines and gracefully shutdown the protected virtual machines. If you notice any errors
during the planned migration, the recovery plan will pause so that you can resolve the errors.
Whereas when you select the option Disaster Recovery as recovery type, SRM will shutdown
any virtual machines at the protected machine, if SRM fails to shutdown the virtual machines
at the protected site, the copies of the virtual machines at the recovery site will still be
started and automatic re-protection for the failed over VM might not be possible.
38
After failover the virtual machines at the recovery site, it is recommended to re-protect the
failed over virtual machines to the protected site to avoid any single point of failure to the
Virtual machines. Re-protection is only possible in the case of protected site storage array is
accessible and functional.
39
After successfully running the recovery plan, the recovery site will start the virtual machine
and connect the virtual machines network assigned in the recovery plan. An example of VM
status is after running a recovery plan is illustrated here. At the recovery site, you can see the
virtual machine Windows-2008-VM1 is up and running and at the protected site shows the
lightning bolt icon in front of the Windows-2008-VM1 virtual machine. This lightning bolt
icon at the protected site indicates the placeholder shadow VM.
The device state for the array pairs windows shows the replication direction state is broken
between local and remote device. The virtual machine at the recovery site is connected with
an appropriate network and accessible to the user.
40
Now lets take a look at the status of the RecoverPoint appliances after running the recovery
plan. It shows that the data transfer is paused and the RPA at remote site has direct access to
the datastore. The protected site of the RPA has no access to the storage.
41
This screen shows the steps executed for re-protecting the VM after failover.
42
When you re-protect the failed over virtual machine after running the recovery plans, RPAs
transfer state becomes active and the data starts following from the recovery sites volume
of the protected sites volume. The protected site ESXi host will not have access to the
volume while the re-protection state is active. The direction state for the local device and
remote device has also changed from broken sign to the reverse direction.
43
You can check the history report for the recovery and re-protecting steps.
44
After performing migration or failover operations for the virtual machines, you have an
ability to failback the VM if the protected site that has been restored to the operational
state. SRM Failback process is to execute the same recovery plan that was used for the
failover but in reverse direction. If you have made any significant changes in the recovery
plan after the VM failover at the recovery site, you must ensure that the protection group at
the protected site is updated accordingly otherwise the failback operations will fail.
The failback steps are executed from bottom to the top order as shown on the screen.
45
A disaster situation happens when there is a site failure and the communication among
systems are not available. The communication failures can be experienced at many levels
such as management network or data replication level. If you are performing a planned
migration then from the recovery site you can bring the VMs on the recovery site from
protection site. In order to simulate the disaster scenario, the datastore access to the VM has
been removed. The datastore unavailability is shown by the gray out snap-735158xxx
datastore.
46
When the VM losses the access to the replicated datastore at the protected site, the
recovery plan gets executed and the failover the VM at the recovery site. You can see the
error message generated during recovery plan execution for example when the recovery plan
is trying to shut down the VM at protected site and preparing the protection group. These
errors will be ignored and ultimately the virtual machine will be started at the recovery site if
the replicated datastore is in consistent stat the recovery site and accessible from the
recovery site ESXi host.
47
This screen shows the status of the virtual machine after the disaster happens at the
protected site. You can see here the folder created on the virtual machines desktop has
consistent data and the virtual machine is up and running on the recovery site.
48
This screen illustrates the RecoverPoint status after the virtual machine is recovered at the
recovery site. As you can see on the screen that the data transfer has been paused between
protected and recovery site and the recovery site has access to the virtual machine.
49
If the protected site configuration failure is rectified and restored to the normal operations, it
is time to re-protect the virtual machine again with the protected site. The moment you
complete the re-protection of virtual machine, you will see the data transfer rate starts
changing and become active.
50
The SRM failback procedure will restore the original configuration of the protected and
recovery site after a recovery. The running a planned migration will bring back the virtual
machines at the protected site from the recovery site and restore the protected and recovery
site relationship.
51
The virtual machine will be started on the protected site and the data transfer will start from
protected site to the recovery site of the SRM configuration.
52
53
54
This lesson covers SRM events and alarms, advanced configurations, and log files collection
for troubleshooting problems during SRM operations.
55
SRM includes a set of alarms specific to SRM-related activity. These alarms function like
other VMware vCenter Server alarms, so you can configure them to send an email, send an
SNMP trap, or run a script. In addition to monitoring SRM tasks, each VMware vCenter SRM
Server monitors the CPU utilization, disk space, and memory consumption of the system on
which it is running. It also maintains a heartbeat with its peer SRM Server.
You can customize thresholds and the way alarms are raised for these events from the
Advanced Settings dialog box. You can get to the Advanced settings by right clicking on the
site name from the SRM window on vSphere Client.
56
VMware vCenter SRM uses the same authorization model as vCenter Server. After you install
and enable the VMware Site Recovery Manager Plug-In, default SRM roles show on the Roles
tab of the Administration view.
SRM includes two sets of roles. Protection roles that allow users to perform protection tasks
at the protected site. Recovery roles that allow users to administer recovery tasks at the
recovery site.
You can also create custom roles by cloning and editing one of the default SRM roles. This
feature allows you to delegate control to a very granular level.
57
The Advanced Settings dialog box provides an easy way for a user with adequate privileges to
change a number of default values that affect the operation of various SRM features.
Changes you make in the Advanced Settings dialog box overwrite the contents of the SRM
configuration file on the SRM Server system.
Use the Recovery page from the Advanced Settings to adjust default values for time-outs
that occur when you test or run a recovery plan.
Use the Storage Provider page to adjust default time-out values and other behaviors of the
SRM SAN provider. The SAN provider is the interface between SRM and your SRA. Certain
SRAs might require you to make changes to default SAN provider values. For more
information, see the SRA documentation from your array vendor.
Use the Local Site Status page to modify default values that trigger alarms monitoring the
SRM Server system at the site to which the vSphere Client is currently connected. You can
change thresholds and the way the related alarms are raised for CPU utilization, disk space,
and memory
consumption. Finally, use the Remote Site Status page to change the criteria and the way
alarms are raised for
a Remote Site Down event.
58
SRM creates certain log files that would be helpful to diagnose the problems. The SRM server
and the client use different log files. The SRM plug-in logs are part of the vSphere client log
file. The SRM log files are collected in a zipped file and placed in the directory (location) you
define during the log collection process. You can also initiate the log collection process fro
the start menu of the SRM server as shown on the screen.
59
This lesson covered SRM events and alarms, advanced configurations, and log files collection
for troubleshooting problems during SRM operations.
60
This module covered prerequisites and configuration of EMC RecoverPoint SRAs.
61
This course focused on installing and configuring VMware vCenter SRM software using EMC
SRAs in a vSphere environment. SRM operations included creation of protection group,
recover plan, and testing recover plan, failover virtual machine, and failback virtual
machines.
62
This concludes the training. Thank you for your participation.
Please remember to complete the course evaluation available from your instructor.
63

RecoverPoint With SRM

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

RecoverPoint With SRM

Hochgeladen von

Copyright:

Verfügbare Formate

This module covers prerequisites and configuration of EMC RecoverPoint.

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Inventory mapping is an options steps but it is recommended. The inventory mapping is

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

It is important to customize the IP address of the virtual machines to maintain its

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved

VMware vCenter SRM Integration with EMC SRAs

Copyright 2012 EMC Corporation. All rights reserved