Beruflich Dokumente
Kultur Dokumente
August 2014
Oracle VM 3:
Change the Server Pool File System &
Repositories (SAN)
White Paper – Change the Server Pool File System & Repositories (SAN)
Contents
Introduction ...................................................................................... 1
Solution Overview............................................................................ 2
Only supported with Oracle VM 3.2.8+ .................................................2
Important terms used throughout the document ...................................2
Goal of the solution...............................................................................3
A note about destroying the server pool ...............................................3
The overall solution at a glance ............................................................3
Key solution highlights explained ..........................................................4
Maintain continuous replication of storage repositories .....................4
Use replicated disks to create a new pool .........................................5
The device special file name will change ..........................................5
The storage repository names remain the same ...............................5
The server pool name remains the same ..........................................6
Process flow .........................................................................................6
Step 1: Preparation Prior to Implementation .................................... 7
Step 1.1: Write down the server pool VIP .............................................7
Step 1.2: Catalog current storage .........................................................7
Step 1.3: Establish continuous replication of storage ............................7
Step 1.4: Perform ad-hoc backup of Oracle VM servers .......................8
Step 2: Release Ownership of Repositories .................................... 9
Step 2.1: Perform ad-hoc backup of Oracle VM database ....................9
Step 2.2: Stop Oracle VM guests running in the pool ..........................11
Step 2.3: Release ownership of current repositories ...........................11
Step 3: Destroy Current Server Pool ............................................. 13
Step 3.1: Remove Oracle VM servers from pool .................................14
Step 3.2: Delete the empty server pool ...............................................15
Step 3.3: Perform final synchronization of storage ..............................16
Step 4: Delete Source Disks .......................................................... 16
Step 4.1: Un-map source LUNs ..........................................................17
White Paper – Change the Server Pool File System & Repositories (SAN)
Introduction
This white paper explains how to change the device special file (DSF) being used as a server
pool file system. This solution can be applied to a variety of situations, but the most common
need for this solution is migrating the pool file system and storage repositories used by a
server pool from a legacy storage array to a new or different storage array. In this case,
system administrators are charged with migrating storage repositories as well as pool file
systems. It is the pool file system that poses the challenge.
Oracle VM 3 does not currently have a built-in tool or way to change the pool file system
without incurring downtime of all running Oracle VM guests. This is because the server pool
must be temporarily destroyed and rebuilt using the new pool file system. There are two major
ways to approach the migration from one pool file system to another.
The first approach presented in this white paper is straightforward with well defined stages that
make it easier for people to follow. This approach involves stopping all Oracle VM guests,
temporarily destroying the server pool and rebuilding the server pool again using a new pool
file system.
Other than this brief explanation, we will not be explaining how to accomplish the second
approach at all since it is a little more complex with slightly blurred stages and doesn’t really
provide any advantage over the first approach. This approach involves using one Oracle VM
server from the legacy server pool to build a new server pool using the new pool file system,
then migrating the remaining Oracle VM servers one-by-one until everything is up and running
in the new pool. You still incur an outage of all Oracle VM guests at various stages using this
method and you will need another server pool VIP, so there is really no advantage to this
process except in cases where you cannot tolerate an outage of all virtual machines during the
same maintenance window.
1
White Paper – Change the Server Pool File System & Repositories (SAN)
Solution Overview
This solution assumes all SAN storage related to a single server pool is being migrated to a new storage
platform including the server pool file system and all storage repositories. In addition, the process of
destroying and then creating a new server pool also changes the OCFS2 cluster ID. This means the
storage repositories will not be available to the new server pool unless these they are first released from
ownership by the Oracle VM Manager.
This document is divided into several parts representing each step in the entire process. The first part
of the document explains the process, process flow, concepts and terms used throughout the
document. The remaining parts of the document are dedicated to accomplishing the steps one-by-one.
! Important!
Please take the time to read and understand the solution before moving directly to the step-by-step process
articulated in this document.
2
White Paper – Change the Server Pool File System & Repositories (SAN)
Target storage: This refers to the new, cloned or replicated PoolFS and storage repositories that will
be used to create the new server pool.
! Important!
Destroying the server pool is not as dramatic as it sounds. Everything related to your Oracle VM guests is
still there for you to use and it is very easy to recover if you follow the instructions in steps 1 and 3.
3
White Paper – Change the Server Pool File System & Repositories (SAN)
This is shown only to explain the differences between the beginning and end of the solution. The
numbered items in each of the illustrations highlight key points that are accompanied by detailed
explanations below.
5 2 1
Oracle VM Servers
Legacy Storage
PoolFS Clone
4 3 RepoFS Clone
RepoFS Clone
Figure 1: Showing the overall state of the storage before beginning the process
5 2 1
Oracle VM Servers
Legacy Storage
PoolFS Clone
4 3 RepoFS Clone
RepoFS Clone
Figure 2: Showing the final overall state of the storage after completing the process
To decrease the downtime associated with the migration process, ensure that you maintain a
continuous replication relationship between the source and target storage platforms until you are ready
to actually stop your Oracle VM guests and release ownership of the storage repositories (see Figure 1,
item 1). You should start the replication process well before you begin the actual migration
documented in this white paper.
4
White Paper – Change the Server Pool File System & Repositories (SAN)
The replication relationship should be maintained and periodically refreshed minimize the delta
between the source and replica storage; this just means periodic replication, not synchronous
replication. For example, you would set up continuous replication between the source and target
storage platforms with scheduled refreshes until the server pool is destroyed in Step 3 of this
document. Notice in Figure 2, item 1 that the replication relationship no longer exists between the
source and target storage repositories after the final refresh has been completed and the old server pool
is destroyed.
The process is completely predicated on replicating/cloning the current LUNs being used for the
current server pool and then completely rebuilding the server pool using the replicated disks. You will
use the current LUNs as shown in Figure 1, item 2 until Step 6 where you build an entirely new server
pool using the replicated LUNs from the new storage platform as shown in Figure 2, item 2.
Compare item 3 in Figure 1 & Figure 2; notice that the WWID (page83 ID) of the device special files
(DSF) for the repository LUNs are different. Our illustration shows a truncated device name showing
only the last four significant digits of the page83 ID – this is simply to help us fit the illustrations on
the page.
During the process you will stop all the Oracle VM guests and destroy the server pool. Destroying the
server pool is a requirement because the DSF name for the PoolFS will be different when you present
the new PoolFS from the new storage platform. Since Oracle VM has no built-in features to simply
change the device special file, we need to destroy the current server pool and create a new one using
the same servers but a different PoolFS. The new PoolFS DSF will have a different WWID.
Notice in both Figure 1 & Figure 2, item 4 that the storage repository names will remain the same
throughout the entire process. The simple name of the repository is contained in a hidden file within
the top level directory named .ovsrepo which will be seen in the Repositories tab of Oracle VM
Manager. Each newly discovered repository will be assigned a random ID by the Oracle VM Manager
as it is added to the management database and will be mounted to /OVS/repositories on the Oracle
VM servers using the new repository ID.
Note
Keep in mind that the Oracle VM guests will not be seen in the new server pool until the target repositories
have been refreshed in Step 9 of this document.
Your Oracle VM guests will remain safe and entirely intact during the entire process. It is normal for
the Oracle VM guests to remain undiscovered in the new server pool until the repositories have been
discovered, presented to Oracle VM servers in the new pool and the repositories refreshed in Step 9 of
this document.
5
White Paper – Change the Server Pool File System & Repositories (SAN)
Figure 2, item 5 shows that you will use the same server pool simple name when you create the new
pool. Item 5 also shows that the newly discovered PoolFS will be assigned a random ID by the Oracle
VM Manager as it is added to the management database and will be mounted to /poolfsmnt on the
Oracle VM servers using the new PoolFS ID.
Process flow
The table of contents shows all the steps for the entire process. However, we will show the following
process flow diagram at the beginning of each step to help you keep track of where you are in the
overall process.
Migrating a Server Pool to a New Storage Platform
6
White Paper – Change the Server Pool File System & Repositories (SAN)
There will be an outage associated with the migration project since you will be destroying and
rebuilding your target server pool You should plan ahead to ensure the outage window is as small as
possible.
The following steps should be performed weeks or days before the planned outage. The choice of
timing to begin the following steps is completely at your discretion since you know how much storage
you will be migrating and how long the preparatory steps might take in your data center given your
operational governance and standard operating procedures.
! Important!
Start your storage replication a day or two before your scheduled outage
7
White Paper – Change the Server Pool File System & Repositories (SAN)
The replication relationship between the source and target storage platform should remain in place
until you destroy the server pool. You will use software on your storage array to accomplish the
storage replication. The following table shows some examples of storage replication software from
various storage vendors.
STORAGE VENDOR LOCAL REPLICATION REMOTE REPLICATION
EMC TimeFinder SRDF
Hitachi Data Systems ShadowImage TrueCopy
Network Appliance Snapshot cloning SnapMirror
Oracle ZFS storage appliance Snapshot cloning Remote replication
If you do not have enterprise class storage with robust replication software, then you can use whatever
standard Linux tools you are comfortable using. The following table shows a few Linux commands
that can be used if you don’t have enterprise class replication tools.
LINUX COMMAND COMMENTS
rsync Recommended: Faster, displays progress of copies and only copies changed blocks on
subsequent writes
cpio Not recommended: Slow, complex to use for remote copies and copies both changed
blocks and unchanged blocks on subsequent writes
dd Not recommended: Slow, complex to use for remote copies and copies both changed
blocks and unchanged blocks on subsequent writes
scp Not recommended: Slow and copies both changed blocks and unchanged blocks on
subsequent writes
The rsync command is probably the best Linux tool to use since it will keep track of changed blocks
between the two copies. The rsync command will only copy changed blocks of data on subsequent
copies of the same repository, so rsync will be much faster for the final synchronization of all storage
repositories during Step 3.
8
White Paper – Change the Server Pool File System & Repositories (SAN)
It should be fine to perform the Oracle VM server backup a day or two prior to your outage, but the
backup of the Oracle VM management database should be performed in Step 2 just before you begin
the outage.
Log onto each Oracle VM server in the target server pool and perform the three tasks show in the
following two screen shots. The first task is to save a copy of the local Berkeley DB on the server as
shown in
The local server side database and OCFS configuration files are the only things that are changed when
you delete or remove an Oracle VM server from a server pool. If you need to back out of the process,
the above files and directories can simply be restored along with the Oracle VM management database
you will back up in the next step.
You must release ownership of the storage repositories before you perform the final refresh or
synchronization of the replicated storage.
9
White Paper – Change the Server Pool File System & Repositories (SAN)
Oracle VM 3.2 and 3.3 have slight variations in the name and location of the backup script. Use the
backup process show in Figure 6 for Oracle VM 3.2:
Figure 6: Oracle VM 3.2 - creating a very quick ad-hoc backup of the Oracle VM management database (MySQL)
Figure 7: Oracle VM 3.3 - creating a very quick ad-hoc backup of the Oracle VM management database (MySQL)
You can use this backup along with the saved files and directories from the Oracle VM servers taken in
the previous step to back out of this entire process.
Note
If you did not choose to use MySQL as the database engine during the install of Oracle VM Manager, then
you will need to contact your Oracle DBA and ask them to back up the database for you.
10
White Paper – Change the Server Pool File System & Repositories (SAN)
2 3 4
Figure 8: Stop all Oracle VM guests running on the target server pool
Stopping the Oracle VM guests is accomplished in the Servers and VMs tab (see Figure 8, item 1).
Notice that Figure 8, items 3 & 4 show our fictional virtual machines stopped and are no longer
assigned to any servers (our example uses Oracle VM guests named myguest51 through myguest56).
Note
You do not need to stop any Oracle VM guests belonging to other server pools being managed by the same
Oracle VM Manager
Stop!
Do not skip this step if SAN storage repositories are part of your migration project. Ownership must be
released BEFORE you begin the final refresh of repositories for later steps to succeed. You may not be able
to complete the migration to new storage if you skip this step.
11
White Paper – Change the Server Pool File System & Repositories (SAN)
The following three screen shots explain how to release ownership of the storage repositories. This
must be completed for all storage repositories that are part of the target server pool. Change to the
Repositories tab as shown in Figure 9 item 1 below, select the first storage repository as shown in
item2 and then choose the Edit Repository icon shown in item 3.
Make sure you check the Release Ownership checkbox as shown in Figure 10 below and then choose
OK to complete.
Ensure the ownership has been released for all storage repositories before moving on to the next step.
If the operation was successful, then you will need to select the Show All Repositories radio button to
12
White Paper – Change the Server Pool File System & Repositories (SAN)
reveal the un-owned repositories (see Figure 11, item 1). Select each repository (Figure 11, item 2) and
ensure that Ownership: indicates Unowned (Figure 11, item 3).
Figure 11: Showing storage repository is no longer owned by the Oracle VM Manager
You will delete the server pool after releasing ownership of the storage repositories. This step is
tedious, but not as dramatic as it sounds. Nothing about Oracle VM is really impacted by this step
other than the server pool won’t exist for a few minutes while we build the new server pool. You will
not need to reinstall or reconfigure anything on the Oracle VM servers; all of the bonding, networking
and anything else about the Oracle VM servers remain unchanged.
In fact, once the server pool is destroyed you can still remount the old storage repositories and start
your VMs by hand without the Oracle VM servers even being part of a server pool if you ever needed
to recover. It is highly unlikely you would need to do such a thing and the only reason we mention this
is to let you know nothing is irreversible once the pool is gone – you still have all the parts and pieces
to rebuild the same server pool and your virtual machines are quite safe.
Note
Destroying the server pool is not as dramatic as it sounds. Everything is still there for you to use and it is
very easy to recover
13
White Paper – Change the Server Pool File System & Repositories (SAN)
Move all Oracle VM servers from the Selected Server(s) box on the right to the Available Server(s)
box on the left as shown in Figure 13 below.
14
White Paper – Change the Server Pool File System & Repositories (SAN)
All Oracle VM servers from the target server pool should now reside in the Unassigned Servers
folder in the navigation pane as shown in Figure 14 below.
Figure 14: All Oracle VM servers from target pool should be in Unassigned Servers folder
15
White Paper – Change the Server Pool File System & Repositories (SAN)
As shown in Figure 16 below, all of your Oracle VM servers from the target pool should be contained
in the Unassigned Servers folder (item 1) and all remaining unaffected server pools should be the
only pools displayed in either the management pane (item 2) or navigation pane.
1
2
Note
Oracle VM is not used in any way to accomplish cloning or replication of storage repositories. Therefore
storage replication is beyond the scope or purpose of this document. We assume you know how to
accomplish the cloning or replication of the storage repositories and physical disks that are part of your
migration project.
Now we will delete the source disks (LUNs) from the Oracle VM management database (model).
Deleting physical disks in this case means deleting the database record for each storage object, not
actually deleting the LUNs that reside on the underlying storage platform. This is a non-destructive
process which means all the data on the LUNs will remain in place and unaffected.
16
White Paper – Change the Server Pool File System & Repositories (SAN)
Deleting the physical disks is an important step since it will help reduce confusion between which disks
are the old source disks and which are the new target disks. You can always rediscover the deleted
disks if you need them again to back out of this solution.
17
White Paper – Change the Server Pool File System & Repositories (SAN)
To delete the disk, simply select all of the disks (item 4) and then choose the Delete Disks icon from
the tool bar above the management pane (item 5).
1 5
4 3
You should no longer see any of the source physical disks once you have deleted them from the model
as seen in Figure 18 below.
Our screen shots assume these are the only physical disks being served from the legacy storage
platform, so you may still see other disks on the legacy storage platform if you are using them for other
server pools that are not part of your migration project.
Figure 18: All of the source physical disks have been deleted
18
White Paper – Change the Server Pool File System & Repositories (SAN)
We deleted the source physical disks from the model simply to reduce confusion over which disks are
which in later steps. We can now discover replicated physical disks from the new target storage
platform.
If you are using Fibre Channel to access your LUNs, then you should simply refresh the Unmanaged
FibreChannel Storage Array to have the Oracle VM servers reprobe the SCSI bus for new LUNs.
This process is documented in the latest Oracle VM user guide for your particular version of Oracle
VM.
19
White Paper – Change the Server Pool File System & Repositories (SAN)
Ensure you are viewing the Storage tab as shown in Figure 19 below (item 1). Select the Unmanaged
FibreChannel Storage Array in the navigation pane (item 2), then choose the Refresh icon from the
tool bar above the navigation pane (item 3).
3 1
You should now see all the replicated physical disks being served from the new storage platform as
shown in Figure 20 below. The LUNs have been catalogued as physical disks in the Oracle VM
management database at this point.
Figure 20: New target disks have now been added to the Oracle VM management database
Note
All Oracle VM servers accessing Fibre Channel disks must be designated as admin servers for the
Unmanaged FibreChannel Storage Array in order to reprobe the SCSI bus on the Oracle VM servers without
a reboot.
Ensure you are viewing the Storage tab as shown in Figure 21 below (item 1). Select the iSCSI SAN
server in the navigation pane (item 2), then choose the Refresh icon from the tool bar above the
navigation pane (item 3).
20
White Paper – Change the Server Pool File System & Repositories (SAN)
3 1
You should now see all the replicated physical disks being served from the new storage platform as
shown in Figure 22 below. The LUNs have been catalogued as physical disks in the Oracle VM
management database at this point.
Figure 22: New target disks have now been added to the Oracle VM management database
We can now rebuild the same server pool using the new PoolFS being presented from the target
storage platform.
21
White Paper – Change the Server Pool File System & Repositories (SAN)
Go to the Servers and VMs tab as shown in Figure 23 below (item 1) and then choose the Create
Server Pool icon from the tool bar above the navigation pane (item 2).
Looking at Figure 24 below, give the new server pool the same name as the old server pool you
destroyed earlier (item 1), enter the same VIP from the destroyed server pool (item 2). Assign a
PoolFS by selecting the Physical Disk radio button (item 3) for Storage Location and then choose
the Search icon to select the new PoolFS (item 4).
3 4
22
White Paper – Change the Server Pool File System & Repositories (SAN)
Make sure you chose the correct SAN server from the Select Physical Disk dialog shown in Figure
25 below. Here we show our fictional iSCSI server; you should choose whatever is appropriate for
your environment.
You should now see the Storage Location field populated with your choice for PoolFS as shown in
Figure 26 below.
23
White Paper – Change the Server Pool File System & Repositories (SAN)
Finally, add all the Oracle VM servers you removed from the old server pool as shown in Figure 27
below.
You have recreated the same server pool at this point using the PoolFS residing on the new storage
platform. There are no additional steps needed at this point other than incorporating the new
replicated storage repositories into the server pool and starting your Oracle VM guests again.
This should be a very easy step as long as you followed the instructions for releasing ownership of the
source repositories before the final replication refresh/synchronization.
24
White Paper – Change the Server Pool File System & Repositories (SAN)
We only need to select the physical disk we know to be storage repositories. Ensure you change to the
Storage tab (item 1) as shown in Figure 28 below. Select the appropriate iSCSI or Fibre SAN server
for your environment (item 2), select all the disks you know to contain replicated repositories (item 3)
and then choose the Refresh Disks icon from the management pane tool bar (item 4).
Figure 28: Refresh the replicated physical disks to update the Oracle VM management database
3
4
25
White Paper – Change the Server Pool File System & Repositories (SAN)
Select each replicated repository in the navigation pane and then choose the Edit Repository icon
(item 1) as shown in Figure 30 below. Use the first sub tab of the wizard to assign each repository to
the correct Server Pool (item 2) and then Take Ownership (item 3).
Use the Present Repositories sub tab in the Edit Repository wizard as shown in Figure 31 below to
present each repository to the newly reconstituted server pool.
Figure 31: Present each repository to the newly rebuilt server pool
Notice that the replicated repositories are now Owned by You (item 1) and presented to all Oracle
VM servers in Figure 32 below. We can now rediscover the Oracle VM guests residing in the
26
White Paper – Change the Server Pool File System & Repositories (SAN)
replicated repositories by selecting the two repositories in the management pane (item 1) and then
choosing Refresh Repositories icon from the tool bar above the management pane (item 2).
Figure 32: All repositories are now owned by the Oracle VM Manager again
The final step is simply to assign the Oracle VM guests to servers and start them.
To start the Oracle VM guests, you will need to select the Unassigned Virtual Machines folder in
the navigation pane as shown in Figure 33 below (item 1). Select one or more Oracle VM guests in the
management pane (item 2) and then choose the Migrate VMs icon from the tool bar above the
management pane (item 3). After that you can start
1 2
27
White Paper – Change the Server Pool File System & Repositories (SAN)
You can drag and drop multiple Oracle VM guests to Oracle VM servers and start them in mass, or
you can migrate and start them one at a time. The point is that you should now be able to run the
same Oracle VM guests from the new replicated repositories.
28
Oracle VM 3: Copyright © 2014, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the
Change the Server Pool File System & contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
Repositories (SAN) warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
August 2014 fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
Document: SN05220 rev. 5 formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
Author: Gregory King means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Oracle Corporation Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and
World Headquarters are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are
500 Oracle Parkway trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0612
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com