Technical Report

Reallocate Best Practices

Chris Wilson, NetApp June 2012 | TR-3929

Summary
The reallocate, read reallocate, and free space reallocate tools offer functionality to optimize the data layout of NetApp storage systems running Data ONTAP . This paper introduces the basic functionality and best practices associated with using the reallocate toolset.

TABLE OF CONTENTS 1 2 Introduction........................................................................................................................................... 3 Reallocate ............................................................................................................................................. 3

2.1 2.2 2.3 2.4 2.5 2.6 2.7 Reallocation Methods...................................................................................................................................... 3 When to Use Reallocate ................................................................................................................................. 4 Reallocate Options .......................................................................................................................................... 4 Execution Requirements ................................................................................................................................. 5 Forced Reallocation ........................................................................................................................................ 5 Developing a Reallocate Schedule ................................................................................................................. 5 Aggregate Reallocation ................................................................................................................................... 6

Continuous Reallocation Features ..................................................................................................... 6

3.1 3.2 3.3 3.4 Read Reallocate.............................................................................................................................................. 6 Read Reallocation and Reallocate .................................................................................................................. 7 Free Space Reallocate.................................................................................................................................... 7 Free Space Reallocate and Read Reallocate ................................................................................................. 7

System and Feature Interactions ........................................................................................................ 8

4.1 4.2 4.3 4.4 Overall System Performance .......................................................................................................................... 8 Volume Snapshot and SnapMirror .................................................................................................................. 8 FlexClone Volumes ......................................................................................................................................... 9 Deduplication and Compression ..................................................................................................................... 9

More Information .................................................................................................................................. 9

Version History ........................................................................................................................................... 9

LIST OF TABLES
Table 1) reallocate start execution options. ......................................................................................................... 4

Reallocate Best Practices

1 Introduction
Data ONTAP provides a set of tools to help customers optimize the layout of data on disk for sequential read access. This guide introduces the reallocate tools in Data ONTAP, including reallocate; the volume option, read_realloc; and the aggregate option, free_space_realloc. This guide also describes some things to consider when using the reallocate tools. The System Administration Guide for the appropriate version of Data ONTAP might answer any additional questions not covered in this guide.

2 Reallocate
Reallocate is used to optimize the layout of data on disk. It works on volumes, LUNs, individual files, and in a special case, aggregates. Reallocate can be started, quiesced, stopped, and scheduled from the command line interface (CLI) by using the reallocate command. You can also use the same command to check the status of the reallocation and measure the current layout optimization. This section describes how reallocate works, when to consider using reallocate, and what the various options perform. Reallocate internally uses the following process: 1. Reallocate measures the current layout and produces an optimization score. If the optimization score is greater than the threshold, proceed to step 2. If the optimization score is less than the threshold, reallocation is not needed. For information about thresholds and measurements, see section 2.2, When to Use Reallocate. 2. Reallocate performs data reallocation by using traditional or physical reallocation. 3. Reallocate rechecks the current layout and repeats the process until the layout is optimal.

2.1

Reallocation Methods

Traditional Reallocation
The reallocation process progresses through the file system and moves data blocks by rewriting them when Data ONTAP determines that the layout can be improved. If no improvement is predicted, no data is moved. NetApp Snapshot data is not moved even when active file system data has been moved to new, optimized locations. Because data is rewritten to disk, if Snapshot copies are used, additional space is required to maintain the copies. The impacts of using reallocate are discussed in section 4, System and Feature Interactions.

Physical Reallocation
The reallocate tool also provides a physical reallocation option. Physical reallocation follows the same process as traditional reallocation; however, instead of completely rewriting data to the disks, the data blocks are moved by changing the physical block location while maintaining the logical block location within the FlexVol volume. The benefit of using physical reallocation is that no additional space is required for Snapshot copies, compared to using traditional reallocation. The implications of using physical reallocation are discussed in section 4, System and Feature Interactions. Best Practice When possible, use physical reallocation to reduce the space requirements of Snapshot copies.

Reallocate Best Practices

2.2

When to Use Reallocate

Reallocate optimizes sequential read performance. The workload that most benefits from the use of reallocate is sequential reads after random writes; however, other workloads might also show some improvement depending on the workload characteristics. Typical applications that see the most benefit from using reallocate include: Online transaction processing databases that perform large table scans E-mail systems that use database storage with verification processes Host-side backup of LUNs

If the workload isnt well known, it might be difficult to determine whether a reallocation is necessary. Reallocate includes measurement functionality to help users quantify if a reallocation is needed. The reallocate measure command provides insight into the current layout by providing an optimization value of the layout as it is when reallocate measure is run. The output values can be used to determine whether on-disk layout is a potential cause of decreased performance. The following command shows how to measure the layout of a volume: toaster> reallocate measure o /vol/volX Note: Use the -o flag to measure layout only and not schedule a measurement. Occasionally, the measurement output might produce a hotspot value. The hotspot value indicates that a portion of the scanned object has a layout that is less optimal than the remainder of the object. For instance, a volume can have an overall measurement value of 3, but contain a hotspot of 28. You can use this optimization value as a threshold setting for reallocate. The default threshold that reallocate uses is 4; you can change this default by using the -t flag when executing a reallocate start or reallocate measure command. Valid threshold values are from 3 through 10 (not optimal). Actual measurement values produced by reallocate can exceed 10. toaster> reallocate start t 5 /vol/volX or toaster> reallocate measure t 5 /vol/volX It is also possible to skip the scanning portion of reallocate by using the n option. This option ignores any thresholds and simply begins data reallocation. toaster> reallocate start n /vol/volX

2.3

Reallocate Options

You can execute and schedule reallocate by using the Data ONTAP CLI. Table 1 describes the available options to the reallocate command.
Table 1) reallocate start execution options.

Option -p

Description Executes reallocate by using physical reallocation. Generally recommended. For more information, see section 2.4, Execution Requirements. Executes a forced reallocation. Generally not recommended. For more information, see section 2.5, Forced Reallocation. Executes reallocation one time only. Executes reallocation without measuring the layout first.

-f -o -n

Reallocate Best Practices

Option -t -A

Description Forces reallocate to use a custom threshold. Executes aggregate free space reallocation.

2.4

Execution Requirements

To run reallocate, you must meet various requirements. The following list can help determine why reallocate might not run: The object to be reallocated must be writable (no SnapMirror destinations). The volume or volume containing an object undergoing reallocation must be at least 5% free. Existing Snapshot copies must leave at least 10% of the Snapshot reserve or 5% of the volume size free, whichever is less. You can use df to check free space. This requirement does not apply to physical reallocation. Full reallocation requires the physical reallocate option (-p) if Snapshot copies are present. Physical reallocate can be executed only on data in FlexVol volumes. Physical reallocate cannot be executed on FlexClone volumes. Physical reallocate (and aggregate reallocate) cannot be run on data stored in mirrored aggregates. Physical reallocate cannot be run on RAID 0 aggregates. Physical reallocate cannot be run on aggregates created with Data ONTAP 7.0 or 7.1.

2.5

Forced Reallocation

A forced reallocation scan optimizes all the data in the volume, LUN, or file regardless of current optimization level. Best Practice Use forced reallocation after adding disks to an aggregate so that existing data is spread onto the new disks, reducing the time required to balance the load across all the disks.

Forced reallocation ignores the optimization thresholds and completely rewrites the data to disk, unlike the normal reallocation process. Although this improves the layout, routine use of reallocate f is not a best practice. Also, because all of the data is optimized, forced reallocation cannot be run against volumes that have existing Snapshot copies unless the physical reallocation method (-p) is also used.

2.6

Developing a Reallocate Schedule

Reallocate is most effective when it is executed regularly. With regular reallocation, the amount of work that reallocate must complete during each iteration can be reduced, and consistently better performance can be achieved. Developing a schedule for reallocate depends on the reallocation method, the time required to reallocate, and the workload applied to the system. By default, executing reallocate start without any parameters causes reallocate to run once daily. To change this default, use the reallocate schedule command after a reallocation scan has been created or use the i flag when creating the reallocate scan. The reallocate schedule command can be used only on reallocation scans that have already been created. A schedule in the form of minute hour day_of_month day_of_week is required as a parameter for the command. An asterisk (*) is a valid

Reallocate Best Practices

option that represents any. The following example creates a weekly reallocate schedule that runs every Saturday at midnight for volX: toaster> reallocate schedule 0 0 * 6 /vol/volX To create a scan with a weekly interval at creation time, use the i flag: toaster> reallocate start i 7d /vol/volX To delete a reallocate job, use the d flag: toaster> reallocate schedule d /vol/volX To stop and delete a job, use the reallocate stop command: toaster> reallocate stop /vol/volX

2.7

Aggregate Reallocation

To optimize an aggregate, use the A option with the reallocate command. This reallocation method reallocates blocks within an aggregate to improve contiguous free space. The A option does not reallocate all of the data in the aggregate following the normal reallocation method. It should not be used to improve sequential read performance. Aggregate reallocation uses the physical reallocation method to move blocks to create contiguous free space, therefore the impacts of using physical reallocate still apply. If you have questions about using aggregate reallocation contact your NetApp representative.

3 Continuous Reallocation Features

The following sections introduce two features in Data ONTAP that perform continuous, on-the-fly data layout optimization: read reallocate and free space reallocate. These features are different from the reallocate command in that they operate continuously without the need to schedule scans.

3.1

Read Reallocate

Read reallocate is a volume option that performs opportunistic reallocation on data to improve performance. Read reallocation uses the normal workload reads along with the read-ahead engine to determine the current layout optimization. If the read was less than optimal, the data will be reallocated to improve the next read of this data. Read reallocate offers both the traditional and physical reallocation methods associated with the reallocate command. Also, because read reallocate uses the existing read workload, it does not require additional scanning or scheduling.

Enabling Read Reallocation

Read reallocate is a volume option that is enabled by using one of the following CLI commands: 7-Mode: toaster> vol options volX read_realloc [on / space_optimized] Cluster-Mode: Cluster::> volume modify vserver vs1 volume volX read_realloc [on / space-optimized] Simply enabling read reallocation by using the on function tells Data ONTAP to use the traditional reallocation method. This has the same side effects as traditional reallocate. The space_optimized option is synonymous with the physical reallocation method. It does not introduce the Snapshot space

Reallocate Best Practices

requirements associated with the traditional reallocation method. The space_optimized method does, however, introduce the Snapshot read performance impact, similar to physical reallocation.

3.2

Read Reallocation and Reallocate

Reallocate and read reallocation are complementary. Both features can be employed together to accomplish the same goal of improving spatial layout. Read reallocate can help reallocate by reducing the amount of work that reallocate needs to do in each iteration, depending on the workload applied to the system. An example of when to employ both is in a database system with weekly large table scans. Enabling read reallocate maintains an optimal layout of frequently accessed data, while a scheduled reallocate execution prior to the large table scan might help improve the table scan speed.

3.3

Free Space Reallocate

Free space reallocation is an aggregate option, introduced in Data ONTAP 8.1.1, which performs opportunistic free space reallocation to maintain an optimal free space layout. When enabled, if Data ONTAP detects that free space is not optimal, it will physically move data to produce areas of contiguous free space. Optimized free space improves the efficiency of WAFL (Write Anywhere File Layout) and can reduce overall disk utilization. Some additional CPU utilization should be expected since Data ONTAP will be doing additional work to manage the movement of data. If the performance of the storage system is limited by CPU, enabling free space reallocation is not recommended.

Enabling Free Space Reallocation

Free space reallocation can be enabled on a per-aggregate basis using the following commands: 7-Mode: toaster> aggr options aggr1 free_space_realloc on Cluster-Mode: Cluster::> storage aggregate modify aggregate aggr1 free-spacerealloc on For best results, the option should be enabled when creating a new aggregate. If enabling the option on an existing aggregate with data already stored on it, there might be a period of time when Data ONTAP will be performing additional work to optimize free space and might affect system performance temporarily. Similar to physical reallocate, aggregate reallocation, and space-optimized read reallocate, free space reallocation will not move blocks that are kept in aggregate Snapshot copies. This does not apply to volume Snapshot copies; free space reallocate will still move blocks stored in volume Snapshot copies.

3.4

Free Space Reallocate and Read Reallocate

When enabling free space reallocation on an aggregate, also consider enabling read reallocate space_optimized for the volumes in the aggregate. The two are complementary technologies that help maintain optimal layout. Read reallocate will optimize the system for sequential reads on the fly, while free space reallocate will optimize for writes.

Reallocate Best Practices

4 System and Feature Interactions

This section describes how the reallocate tools interact with other features in Data ONTAP.

4.1

Overall System Performance

By design, reallocate has a minimal impact on system performance. If a system is heavily loaded, reallocate is treated as a low-priority operation and should not significantly slow users access to data. However, a heavily loaded system also causes reallocate to take longer to complete. Best Practice When possible, schedule reallocate runs during off-peak hours.

4.2

Volume Snapshot and SnapMirror

Both the traditional and physical reallocation methods affect existing Snapshot copies in some way. The following two subsections describe how Snapshot copies and SnapMirror are affected by both methods.

Traditional Reallocate
As previously described, the traditional reallocation method rewrites the data in the active file system to improve layout on disk. Snapshot copies do not require any additional space unless a block in the active file system is altered or removed. Because reallocate rewrites data to improve layout, a nonoptimal file system layout requires many data blocks to be rewritten. WAFL improves the layout by moving the data blocks, but the original data block remains in any existing Snapshot copies, thus increasing the required Snapshot space. Administrators must plan for this additional capacity utilization when running reallocate. The additional utilization lasts only as long as the life of the Snapshot copies. SnapMirror hinges off of Snapshot technology to create point-in-time mirroring. When using the traditional reallocation method against a SnapMirror source, SnapMirror flags any reallocated data as data that needs to be sent during the next update, regardless of whether the contents have actually changed. Therefore, if a large amount of data was reallocated, SnapMirror will require a large update on the next sync.

Physical Reallocate
The physical reallocation method does not have the same impact on Snapshot copies as the traditional reallocation method. Because physical reallocation does not follow the complete rewrite process, but rather moves data blocks at a lower level without altering the logical location, Data ONTAP Snapshot technology does not notice any changes. This low-level movement does not cause the copies to occupy any additional capacity. Using physical reallocate can introduce a read performance impact to existing Snapshot copies. This performance impact is due to additional operations that must be completed to locate Snapshot data after a reallocation completes. After a block is read from a Snapshot copy, subsequent accesses to that block do not require additional lookups. Physical reallocation does not affect the performance of copies created after reallocate completes. Physical reallocate does not cause the active file system to diverge from existing Snapshot copies. Unlike the traditional reallocation method, SnapMirror does not identify any data that was reallocated as data that must be sent during the next sync.

Reallocate Best Practices

4.3

FlexClone Volumes

FlexClone volumes also hinge off of Snapshot technology, so executing reallocate has similar implications. Running reallocate against a parent causes any data that is reallocated to be maintained in the Snapshot copy for the clones, using additional Snapshot reserve space. If the clone is reallocated, it diverges from the parent and uses additional space in the aggregate. The parent remains in the same, unoptimized state. If physical reallocate is used against a parent of FlexClone volumes, the parent is reallocated as expected; however, the clones might exhibit some read performance degradation during the first access of data due to additional location lookups. Physical reallocate and space_optimized read reallocation against a clone are not allowed.

4.4

Deduplication and Compression

Starting in Data ONTAP 8.1 deduplicated data can be reallocated using physical reallocation or read_realloc space_optimized. Although data may be shared by multiple files when deduplicated, reallocate uses an intelligent algorithm to only reallocate the data the first time a shared block is encountered. Prior versions of Data ONTAP do not support reallocation of deduplicated data and will skip any deduplicated data encountered. Compressed data will not be reallocated by reallocate or read reallocate, and it is not recommended to run reallocate on compressed volumes.

5 More Information
To get more information about using reallocate, read reallocate, or free space reallocate, review the System Administration Guide on the NetApp Support site for your version of Data ONTAP.

Version History
Version
Version 1.0 Version 1.1 Version 1.2

Date
June 2011 October 2011 June 2012

Document Version History

Initial publication. Updated for 8.1.0 release, including deduplication changes. Updated for 8.1.1 release, including free_space_reallocate.

Reallocate Best Practices

Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customers responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.

10

2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, Data ONTAP, FlexClone, FlexVol, SnapMirror, Snapshot, and WAFL are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as Reallocate Best Practices such. TR-3929-0512