Volume 19 | Number 3 Third Quarter 2012

w w w . i o u g . o r g

F o r t h e C o m p l e t e Te ch n o l o g y & D a t a b a s e P r o f e s s i o n a l

Security
Oracle Enterprise Manager 12c Infrastructure and Operational Security
by Janet Wakeley

Resolving Child Cursor Issues Resulting in Mutex Waits

by Martin Klier

Application Development with Oracle Advanced Queuing

by Jeffrey Jacobs

Basic server architectures havent changed for decades. Thats why Cisco introduced the Cisco Unified Computing System (UCS)which integrates compute, high-speed networking, storage access and virtualization. Since its introduction just over 2 years ago, 10,000 customers have adopted Cisco UCS as their server architecture. For Oracle users Cisco UCS has set multiple Oracle performance benchmark records and is helping Oracle users dramatically reduce data center complexity while: Delivering accelerated Oracle performance. Reducing Oracle deployment times from weeks to minutes Lowering operating costs by up to 30%. Providing implementation flexibility to support non-virtualized and virtualized Oracle installations The Cisco Unified Computing System, powered by intelligent Intel Xeon processors, introduces new architecture to Oracle environmentswhere everything, and everyone, works together like never before. Find out more at www.cisco.com/go/ioug.

Visit Cisco at Collaborate 12 in Las Vegas

2012 Cisco Systems, Inc. All rights reserved. All third-party products belong to the companies that own them.

10,000 customers have switched to Cisco servers

Executive Editor April Sims Associate Editor John Kanagaraj Asia-Pacific Technical Contributor Tony Jambu Managing Editor Theresa Wojtalewicz Associate Editor Alexa Schlosser Contributing Editors Ian Abramson Gary Gordhamer Michelle Kolbe Arup Nanda Ray Smith Board Liaison Todd Sheetz How can I contribute to SELECT Journal? Write us a letter. Submit an article. Report on Oracle use in all corners of the globe. We prefer articles that conform to APA guidelines. Send to select@ioug.org. Headquarters Independent Oracle Users Group 401 North Michigan Avenue Chicago, IL 60611-4267 USA Phone: +1.312.245.1579 Fax: +1.312.527.6785 E-mail: ioug@ioug.org Editorial Theresa Wojtalewicz Managing Editor IOUG Headquarters Phone: +1.312.673.5870 Fax: +1.312.245.1094 E-mail: twojtalewicz@ioug.org How do I get the next one? SELECT Journal is a benefit to members of the Independent Oracle Users Group. For more information, contact IOUG Headquarters at +1.312.245.1579 SELECT Journal Online For the latest updates and addendums or to download the articles in this and past issues of SELECT Journal, visit www.selectjournal.org.
Copyright Independent Oracle Users Group 2012 unless otherwise indicated. All rights reserved. No part of this publication may be reprinted or reproduced without permission from the editor. The information is provided on an as is basis. The authors, contributors, editors, publishers, the IOUG and Oracle Corporation shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this publication or from the use of the programs or program segments that are included. This is not a publication of Oracle Corporation, nor was it produced in conjunction with Oracle Corporation.

CONTENTS
Volume 19, No. 3, 3rd Qtr. 2012

Features
5 Oracle Enterprise Manager 12c Infrastructure and Operational Security By Janet Wakeley OEM components require security considerations including authentication, authorization and encryption to ensure proper integrity and availability. Janets article takes a look at the various components and the options for securing them and provides recommendations based on best practices and business constraints.

13 Resolving Child Cursor Issues Resulting in Mutex Waits By Martin Klier Martin discusses how mutex waits are now seen more often in Oracle Database 11gR2 and delves into some of the basic concepts related to parsing and serialization. 20 Application Development with Oracle Advanced Queuing By Jeffrey Jacobs Jeffrey offers an article about understanding the basic features and capabilities of Oracle Advanced Queuing (AQ) as it applies to application development. 25 Time for Change: Migrate your Non-RAC Database to RAC By Satishbabu Gunukula Satishbabus article contains the basic steps for converting a single-instance database to Real Application Clusters. It also includes a list of recommendations, best practices and tips from an accomplished professional. 29 Use Case Comparison of Oracle Database 11g on Solid State Disks (Part Two) By Mich Talebzadeh This is the second article by Mich covering the specific use case of solid state disks for Oracle 11g tablespaces for write-intensive operations. His first article on read-intensive activities can be found in Q2 2012.
Reviewers for This Issue
Greg McEachern Andy Rivenes Lee Murray

Regular Features
2 From the Editor 2 From the IOUG President 28 Users Group Calendar 33 Advertisers Index 34 SELECT Star 36 Oracle ACE Spotlight

3rd Qtr 2012 Page 1

From the Editor

From the IOUG President

Welcome to the Q3 2012 issue A Promising Future of SELECT Journal

John Matelski IOUG President April Sims Executive Editor
This issue is distributed at Oracle OpenWorld, Oracles annual conference in San Francisco. What I have seen throughout the years attending OOW is the extensive amount of networking, interaction and discussion between Oracle employees and partners and their customers. It is important to be part of these types of discussions if you want to influence the future of an Oracle product. This edition of SELECT Journal gives a nod to security. A key piece related to this subject is Janet Wakeleys article, Oracle Enterprise Manager 12c Infrastructure and Operational Security. While there are even more secure features built into OEM 12c than previous versions, it is up to an implementer to properly maintain and audit the security aspects of 12c. Look for another security-related article by Wakeley on Oracle Audit Vault in an upcoming issue. This issues Oracle ACE Spotlight wherein SELECT Journal highlights certain IOUG members features Michelle Malcher. Malcher was chosen because she is a security-focused leader in her field. Expect to see a personal glimpse into what has been important in her professional career. I am truly honored to serve in my new role as the organizations president. It is an exciting time for IOUG, with great momentum being created from a very successful COLLABORATE 2012. We enjoyed a 10percent increase in conference attendance compared to last year, and the satisfaction ratings were very high. With such a great conference behind us, its now time to look ahead to what IOUG has to offer you in the coming year. Here is a brief outline of my goals: 1. Build upon the many successes that have already been achieved. Illbe working with the IOUG board throughout the year to review our strategic direction and offerings. Where it makes sense, we will explore possible changes to increase the value of membership with IOUG. 2. Continue to maintain and enhance our relationships with Oracle. We will always be an independent user group, with no direct ties to Oracle or any financial dependence on them. But we do value the connections we have with Oracle, and it is important that we continue to be their most valued technology user group partner. 3. Enhance our offerings to meet the evolving needs of the user community. As the technology we use continues to change, our needs as professionals will change, and the composition of our community will also change. We will continue to keep our pulse on the needs of our evolving community, and we will modify and enhance our educational content, delivery methods and our other services to maximize value. 4. Continue to expand and provide opportunities to our volunteers. Volunteers are at the heart of everything that is great about IOUG. Its professionals like you who give their time to present at conferences, organize and deliver our offerings, and keep our voice strong with Oracle. We will make sure that many opportunities exist for you to play leadership roles that enhance your careers and increase the value of IOUG to the entire community. I am thrilled for what the coming year has in store. Our new Exa-Everything program is a wonderful example of our partnership with Oracle and will be a great way for you to learn about this valuable technology. We also have some excellent webinars planned for the summer and fall. I encourage all of you to become involved in one or more of our special interest groups (SIGs). You can learn more about these and other offerings by visiting www.ioug.org. I hope you are looking forward to the rest of 2012 and the coming of 2013 with the same level of enthusiasm that I am. I invite you to take advantage of all that IOUG has to offer and become involved in whatever way fits best with your personal interests and professional goals. Sincerely,

2013 Themes and Topics We are actively looking for articles, features and discussions related to the following topics for the last issue of this year themed Business Intelligence/ Data Warehouse: big data, performance, advanced features and case studies. See the next issue for the announcement of 2013 themes for SELECT Journal.
Once again, we would like to thank all our authors and reviewers for their efforts in providing high-quality content for the journal. We always welcome your input and feedback, and we especially look forward to you sharing your expertise via this fine medium. Email us at select@ioug.org if you would like to submit an article or sign up to become reviewer.

April Sims Executive Editor, IOUG SELECT Journal

John Matelski IOUG President

Page 2 3rd Qtr 2012

T Like A Pro une

DB Optimizer makes database performance tuning easy

Optimize SQL faster using Visual SQL Tuning Identify performance bottlenecks immediately Simplify database management with one tool Visit www.embarcadero.com/optimize for more information

Oracle Enterprise Manager 12c Infrastructure and Operational Security

Attribute
Confidentiality Integrity Availability

Threat

Protection

data breach; man-in-the-middle encryption of data at rest and attacks in-transit; access controls data modifications data inaccessible; DoS (denial of service) least privilege authorization and approval; code testing and migration supported platforms; patching

Table 1 IT security controls are often implemented without a proper risk analysis being completed. The result is an all-or-nothing approach, where the controls can either far exceed adequate protections or fall drastically short of preventing even the most common attacks. It is important to understand not only the components and communication that make up the system, but also the environment in which it is being deployed and the targets it is managing to do an accurate risk assessment. Risk is the likelihood that potential threats may exploit vulnerabilities. It is important to note that the loss potential of some risks may not warrant the cost of protecting against it. Also, the degree of risk may vary greatly depending on the environment (e.g., managing hosted customer applications in the cloud versus one used to manage top secret government document repositories).

By Janet Wakeley Edited by Ray Smith

nterprise Manager Cloud Control (EM 12c) is system management software that provides centralized monitoring, administration and life-cycle management functionality. The operational control and infrastructure components of the tool require security considerations including authentication, authorization and encryption to ensure proper integrity and availability. This document will take a look at the various components and the options for securing them and will provide recommendations based on best practices and business constraints.

Security Overview A defense-in-depth approach to securing the EM 12c infrastructure ensures that all layers of the technology stack are evaluated for potential risks and secured appropriately. At the same time, it is important to avoid impeding the productivity gains afforded by the tool with an overzealous or unsuitable locking down of the environment. It is important to understand the threats and vulnerabilities so appropriate countermeasures can be deployed.
Confidentiality, integrity and availability also known as the security triad are the key attributes to keep in mind when identifying and applying security standards to hardware, software and communications that compose an information system. Confidentiality is the term used to describe the prevention of information disclosure to unauthorized individuals or systems; integrity means that data cannot be modified undetectably; and for any information system to serve its purpose, the information must be available when needed. Table 1 is a high-level overview of typical security concerns and some of the protections used to mitigate the risk of a vulnerability being exploited. It is not meant to be an exhaustive list but, rather, provide a quick snapshot of some common information system security concerns and mitigating controls.

Figure 1: Enterprise Manager Components and Communication Links Figure 1 is a simplified diagram showing the main components of a typical Enterprise Manager 12c configuration and the communication paths between them. The diagram shows six main components:

HTML Console Web Logic Server (WLS) hosting the Oracle Management Service (OMS) Oracle Database Repository Target agent(s) Target database(s) and other managed target technologies Oracle Corporation

continued on page 6
3rd Qtr 2012 Page 5

Oracle Enterprise Manager 12c Infrastructure and Operational Security continued from page 5

There are five critical communication links:

OMS Console OMS Target agent(s) Including agents deployed on the OMS and Repository DB Host OMS Target database(s) Including the Repository DB OMS Oracle Corporation (My Oracle Support) OMS Other managed Target Technologies
The following sections review the components, the communication links between them and offer recommendations for securing them.

of confidentiality, integrity and availability and should be considered carefully and managed appropriately. First, the machines that host the web tier, application tier and repository database should be hardened. For specific recommendations and how-to instructions for various platforms, refer to the Center of Internet Security (CIS) benchmarks.(2) As an example, the benchmark for RedHat Linux 5 mandates installing updates, applying security patches, removing unsecure services such as rsh, rlogin, telnet, etc., and minimizing the attack surface by not installing, removing or disabling extraneous components or services. These hardening steps have the additional benefit of freeing up resources that can be used to improve the performance and availability of the OMS. Other recommendations include filesystem and network configurations, enabling auditing and logging, and implementing strong authentication and authorization controls. In addition, Oracle recommends protecting the WebLogic Server (WLS) home directory, especially the domain directory, which contains configuration files, security files, log files and other Java EE resources for the domain. This protection is achieved by limiting the OS users who have access to the WLS home directory and limiting the number of system administrator accounts. Patching all Oracle homes is also strongly recommended, including the OMS, repository DB, agents and managed targets with the latest Critical Patch Update (CPU) patches. Patching ensures known vulnerabilities within these applications are fixed and further reduces the attack surface to prevent common exploits such as SQL injections, privilege escalations and denial-of-service. There are additional compliance standards for database and WLS that can be applied depending on the level of security required. Refer to the Oracle Fusion Middleware Securing a Production Environment for Oracle WebLogic Server (3) and the Oracle Database Security Checklist (4) for more detailed information.

HTML Console The HTML Console is used by administrators to access the Oracle HTTP server hosting EM 12c. Typically, this would be PC type hardware using browser software such as Microsoft Internet Explorer or Mozilla Firefox and communicating remotely to and from the server hosting the EM 12c application. As such, admin messages could be intercepted, spoofed or falsified. Oracle provides the option to use the unsecured HTTP protocol or the encrypted HTTPS (SSL/TLS over HTTP) protocol. The latter is the default in EM 12c. If stricter rules are required, a firewall can be used to further restrict access.
Another key consideration is that some data from EM 12c can be downloaded to the remote PC, such as inventory reports, configuration data and target compliance details. Therefore, it may be necessary to take precautions to secure the PC against data loss, such as encrypting the hard drive, strong authentication and access controls. If administrators work outside of your enterprise network, a VPN should be used to secure the connection. EM 12c has the ability to use an iDevice to remotely connect to Enterprise Manager for the purpose of managing incidents and problems using the Cloud Control Mobile app available from the iTunes store. This app supports connections over VPN.

Oracle Management Service The Oracle Management Service (OMS) receives monitoring data from the various agents and loads it into the management repository. The management console retrieves data from the management repository, organizes it and then displays it as information to the administrator via the HTML console interface. Plug-ins are entities that offer management capabilities for specific target types. They work together with the OMS and the management agent to monitor targets in the environment. The mandatory management plug-ins include the following:

Oracle Management Service Communication It is imperative to secure the communication between the Oracle Management Service and the various components it interacts with to ensure sensitive data such as credentials and other target data are transmitted securely.
In EM 12c, the default installation will automatically secure-lock the OMS, which requires that agents communicate only through HTTPS port and that the HTTP port is locked. This ensures that communications are always encrypted and mutually authenticated. All requests from un-secure agents are rejected by the OMS and any un-secure request from the OMS is rejected by the agent. This helps safeguard the management system from man-in-themiddle attacks occurring from within the infrastructure. To ensure the console access from the client browser is secure over SSL/TLS, the console must be locked, as well. To enable Enterprise Manager Framework Security for the Management Service, use the emctl secure OMS utility to perform the following actions:

Oracle Database used to monitor and manage Oracle Database and

related targets such as RAC and ASM.

Oracle Fusion Middleware used to monitor and manage Oracle Fusion

Middleware products such as Oracle WebLogic Domain, Oracle WebLogic AdminServer, Oracle WebLogic Server, Oracle SOA Suite and Oracle WebTier. console, search the knowledge base, raise service requests and create patch plans and templates for patching monitored targets.

Generate a root key in the management repository. The root key is used

Oracle MOS used to log in to My Oracle Support from within EM 12c

during distribution of Oracle wallets containing unique digital certificates for management agents. service and management agents, independent from any existing HTTPS configuration that may be present in the WebTier. agents using Enterprise Manager Framework Security.

Modify the WebTier to enable an HTTPS channel between the management Enable the management service to accept requests from management

The OMS is the heart of the management system and is used to monitor and manage all the various supported targets. In order to do so, it must be allowed to connect to those targets and perform a broad set of privileged administrative activities. The security implications of this involve all aspects

Page 6 3rd Qtr 2012

Note that once OMSs are running in secure-lock mode, unsecure agents will not be able to upload any data to the OMSs. Agents should be deployed using the EM 12c agent deploy job, which uses the secure SSH protocol. If manually deploying agents, protect against the possibility of unauthorized agents being installed by using one-time registration passwords that have a reasonable expiry date instead of persistent registration passwords. The OMS can be configured in the following modes:

without it, OMS will fail to start. By storing the key separately from the Oracle Enterprise Manager schema, the sensitive data in the repository remains inaccessible to the schema owner and other SYSDBA (highly privileged) users.

TLSv1-only mode SSLv3-only mode Mixed mode

Mixed mode is configured by default in EM 12c, but Oracle recommends configuring the OMS in TLSv1 only mode as a security best practice. This difference may be due to the differences between the hash functions used for the master key derivation process and that, as such, the SSL 3.0 implementations cannot be validated under the Federal Information Processing Standards (FIPS-140-2). This is not to say that TLSv1 is without security concerns. In September 2011, researchers used a Java Applet to violate same origin policy constraints for a long-known cipher-block-chaining (CBC) vulnerability referred to as BEAST (Browser Exploit Against SSL/TLS). Mozilla Firefox, Google Chrome and Microsoft have all developed and released fixes to mitigate BEAST-like attacks. It can also be prevented by removing all CBC ciphers from the list of allowed ciphers. The key takeaway here is that the cat and mouse game between security professionals and attackers continues and should be part of the risk equation when evaluating the level of control necessary to secure the environment. This also highlights the need to look at security in a holistic view; all levels of the technology need to be updated with relevant fixes in order to prevent breaches. Often desktops and browsers are overlooked, in combination with the application server and all Java components.

Repository DB Communication The Oracle Net foundation layer establishes and maintains the connection between the client application and database server using Transparent Network Substrate (TNS) technology, which enables peer-to-peer application connectivity. The Oracle protocol support layer is responsible for mapping TNS functionality to industry-standard protocols used in the client/server connection. The default protocol used by the repository database is TCP/IP, which means data will be transmitted in clear text. In order to encrypt the data, the Oracle Advanced Security Option (ASO) is required to enable TCP/IP with SSL and is a separately licensed product provided by Oracle.
The credentials stored in the EM 12c repository DB are encrypted by the emkey and are protected. Other data stored in the repository consist of items such as infrastructure configurations, inventory, performance metrics and compliance metrics and, as such, may not warrant the cost associated with additional security controls. Note that if EM 12c is used to manage databases containing classified or regulated data, administrators have been granted access privileges to that data, and the data is not encrypted, it could be transmitted in clear text. If the repository DB is the only database on the host, you can restrict network access to the repository DB host by configuring the Oracle listener to only accept requests from OMS nodes by adding the following parameters in the TNS_ADMIN/protocol.ora file:

tcp.validnode_checking =YES tcp.excluded_nodes = (list of IP addresses) tcp.invited_nodes = (list of IP addresses of OMS nodes)
Advanced Security Option Oracle Advanced Security Option (ASO) ensures the security of data transferred to and from an Oracle database. As previously stated, it is a separately licensed product that can be configured to improve the security of both the EM 12c infrastructure and any target Oracle databases that may benefit from the security enhancements that ASO provides. See the Oracle Database Advanced Security Administrators Guide 11g Release 2(7) for more details. If ASO is enabled for the repository DB, the OMS and agent must also be configured to connect to a secure database. Firewalls Firewalls protect a companys IT infrastructure by providing the ability to restrict network traffic, examining network packets as they arrive and determining the appropriate course of action. Firewall configuration typically involves creating rules to restrict the ports and protocols that can be used on the network. Most organizations use firewalls, and it is important to understand how to deploy Enterprise Manager 12c components with firewalls to gain the maximum benefit of protection while minimizing any potential negative impact they may have on the effectiveness of the tool. Oracle suggests that firewall configuration should be the last phase of EM 12c installation, or, if you are installing in an environment already using firewalls, that the default EM 12c ports be open for all traffic until you complete the installation, configuration and verification. After completing the EM 12c installation, you can view the ports assigned in the staticports.ini file on the OMS host located in MIDDLEWARE_HOME/.gcinstall_temp/. Default port values for various components are shown in Table 2 on the next page:
continued on page 8
3rd Qtr 2012 Page 7

Repository DB The Oracle Enterprise Management Repository database stores information collected and used by the management agents. It consists of objects such as database jobs, packages, procedures, views and tablespaces. As previously stated, all security best practices for securing Oracle databases and the hosts they reside on apply to the repository database as well. Special consideration should be given to the highly privileged accounts used to manage EM 12c. SYSMAN is the schema owner, and access to this account should be secured following similar procedures used to secure other privileged accounts (SYS, SYSTEM) in the environment. Users should use individual accounts and be granted only the privileges required to perform the tasks assigned them. More information regarding authentication, authorization and audit capabilities will be discussed in detail in the operational security section of this document.
Credentials used to access hosts, databases and application servers are stored in the repository DB and may include basic username-password or strong authentication schemes such as PKI, SSH-keys and/or Kerberos. Since the credentials are stored in the repository DB, it is critical they be stored and transmitted securely. Oracle EM 12c provides cryptographic support of sensitive data by a sign-on verification method known as the emkey. This encryption key is the master key used to encrypt/decrypt sensitive information such as passwords and preferred credentials that are stored in the repository DB. The key is originally stored in the repository, then removed and copied to the credential store during installation of the first OMS. A backup of this file should be generated and stored securely;

Oracle Enterprise Manager 12c Infrastructure and Operational Security continued from page 7

Port
Enterprise Manager Upload HTTPS (SSL) Port Management Agent Port Management Repository Database Port Cloud Control Console HTTPS (SSL) Port EM Domain WebLogic Admin Server HTTPS (SSL) Port Cloud Control Managed Server HTTPS (SSL) Port WebLogic Node Manager HTTPS (SSL) Port

Default Values
1159, 4899 - 4908 3872 1521 7799 -7809 7101 - 7200 7301 - 7400 7401 - 7500

ports. Because the network will be scanned, the sudo privilege delegation must be set on the management agent host that will perform the scan. Typically, the management agent installed on the OMS host is used to perform the scan. Notification System The notification system allows EM 12c administrators to be notified when specific incidents, events or problems arise. It also has the capability to perform actions such as executing OS commands, scripts and PL/SQL procedures based on specific incidents or events. The notification system can also send traps to Simple Network Management Protocol (SNMP) enabled third-party applications. Typically, notifications are sent via email. Before Oracle Enterprise Manager can send email notifications, an outgoing mail (SMTP) server must be set up for use by the notification system. Setup parameters include one or more outgoing mail server names, the mail server authentication credentials, the sender name for notifications and the email address to be used. The notification system can optionally be configured to use secure connections by entering SSL in the secure connection field. Email is encrypted using the TLS protocol if the mail server supports it; otherwise, the email is automatically sent as plain text. Software Library Oracle Software Library is a repository that stores software entities such as software patches, virtual appliance images, reference gold images and application software. In addition to storing them, it can be used to maintain versions, maturity levels and life cycles of these software entities. The software library leverages the organizations existing IT infrastructure (file servers, web servers and storage systems) to stage the files to host targets as part of provisioning or patching activity. Two types of locations are available: an upload file location and referenced file locations. For single OMS environments, the software library can be configured on either the host where the OMS is running, or in a shared location. With EM 12c, a new feature called referenced file location has been introduced, wherein software library entities can refer to files that are stored on another host. These locations are read-only for the software library and are not used for uploading files. Referenced file locations support three storage options: 1) HTTP: an HTTP storage location that represents a base URL, which acts as the source of files that can be referenced 2) NFS: an NFS storage location that represents an exported file system directory on a server 3) Agent: An agent storage location can be any host monitored by an Enterprise Manager Agent. The agent can be configured to serve the files located on that host. From a security point of view, the software library should be protected against unwanted alterations to the source patches and software images. Adequate space should be made available for the library to store and manage the essential software used in the environment being managed by EM 12c. Uploads and other maintenance activities are supported through the credential subsystem. Note that if shared storage or referenced storage is utilized in an environment where firewalls exist between the OMS and targets, it is important to set the rules to allow appropriate communication between the shared storage location(s) and the OMS. Information Publisher Reports EM 12cs reporting framework, Information Publisher, makes information about the managed environment available to audiences across your enterprise. Reports are used to present a view of monitoring information for

Table 2 To configure the management agent installed on a host protected by a firewall to communicate with an OMS on the other side of the firewall, perform the following tasks:

Configure the management agent to use a proxy server for uploads to the
management service.

Configure the firewall to allow incoming HTTP traffic from the

management service on the management agent port. To configure the OMS installed on a host protected by a firewall to communicate with management agents on the other side of the firewall, perform the following tasks:

Configure the management service to use a proxy server for its

communications to the management agents.

Configure the firewall to allow incoming HTTP traffic from the

management agents on the management repository upload port.

Access to My Oracle Support from within EM 12c provides customers with the ability to search the knowledge base, raise service requests and create patch plans for managed targets. Mean time to resolution (MTTR) can be greatly reduced by providing in-context information specific to the customers infrastructure by uploading configuration data on a regular basis. If internal security policies permit, Oracle Management Server may be enabled to access My Oracle Support by making the following URLs available through the firewall: ccr.oracle.com, login.oracle.com, support.oracle.com, and updates.oracle.com and setting up a proxy server so EM 12c can access My Oracle Support. Be sure to review the Oracle Configuration Manager Security Overview(15) documentation to understand the information that is collected and how it is secured by Oracle Corporation.

Other Security Considerations ICMP Oracle Management Service uses the Internet Control Message Protocol (ICMP) Echo Request to check the status of target host machines. If the ICMP Echo Request is blocked by a firewall, the target host machine will appear to be down. Enable the ICMP Echo Request so that the ping command can be issued by the OMS to check the status of the machine. The default port (7) is used for the ICMP Echo Request.
Auto-Discovery of Targets Another feature of EM 12c that may be impeded by certain firewall configurations is the ability to auto-discover targets. In automatic host discovery, a single management agent is tasked to scan the entire network based on IP address ranges. It then returns a list of unmanaged host machines, a list of ports in use and the name of the service using each of the

Page 8 3rd Qtr 2012

business intelligence purposes. The reporting framework allows users to create and publish customized reports via the web or email reports to selected recipients. EM 12c renders a separate reporting website that does not require user authentication. To ensure that no sensitive information is compromised, a special system privilege must be granted to those administrators that are allowed to publish reports to the Enterprise Manager reports website. As a minimum, the administrators granted this privilege should be educated on the data classification and internal policies governing those classifications.

When using an authentication scheme based on Oracle Internet Directory

(OID) as the identity store, plug in the OID-based authentication scheme to allow applications to authenticate users against the OID. scheme to authenticate users against the Microsoft Active Directory.

When using a Microsoft Active Directory as an identity store, plug in this

Authenticating EM users to targets Credentials are typically required to access targets such as databases, application servers and hosts. As detailed in the infrastructure security section, credentials are encrypted using emkey and stored in the repository DB. Beginning with Enterprise Manager 12c, the credential subsystem supports strong authentication schemes such as PKI, SSH-keys and Kerberos. The use of PKI and Kerberos require Oracle Advanced Security Option licensing for all the Oracle database targets where it is deployed. SSH-key based host authentication, used by jobs, deployment procedures and other EM 12c subsystems, is now also supported. Credentials can be classified into the following categories:

Operational Security Authentication, authorization and auditing work together to provide EM 12c operational security by ensuring the identity of the users, controlling access tosecure resources and functions, and establishing a high assurance of nonrepudiation. The following sections explain the EM 12c features used within these areas to provide robust operational system security.
Authenticate The most common form of authentication is login credentials and passwords and is the default authentication method for access to EM 12c. If using this method, care should be taken to make sure a strong password policy exists and is systematically enforced to protect against password cracks or brute force attempts. The most highly privileged account is the SYSMAN account. This account owns the underlying schema and objects of the repository DB and should be secured by limiting its use and the disclosure of its password to only the tool administrator. The EM 12c authentication framework covers both the authentication to the EM 12c console and authentication to the managed targets. The tasks an administrator can perform and targets they can access depend on the roles, system privileges and target privileges granted. The super administrator can choose to let certain administrators perform only certain management tasks, access only certain targets or perform certain management tasks on certain targets. In this way, the super administrator can provide a separation of duties among the administrators. Authentication to EM 12c EM 12cs authentication framework consists of pluggable authentication schemes that support authentication protocols that may already be in use. Aside from repository-based authentication, the other options listed here are separate licensable products:

Named credentials Job credentials Monitoring credentials Preferred credentials

Named Credential Administrators define and store credentials within EM 12c as a named entity. Named credentials can be a username/password or a public key-private key pair and are used for performing operations such as running jobs, patching and other system management tasks. There are two categories of named credentials: global-named credentials and target-named credentials. Global-named credentials are independent entities not associated with any objects at the time of creation, whereas target-named credentials are associated with individual targets when they are created. Access control for credentials ensure that only credential owners can grant privileges on their created credentials to other users, and the credentials owned by an administrator will be deleted if that administrator is deleted from EM 12c because sharing ownership is not allowed. There are three privilege levels for all credentials: view, edit and full. Full allows an administrator to edit all details, including the authentication scheme, and delete the credential. Edit allows an administrator to change the password or public/private key pair but not the name of the credential or the authentication scheme. View allows the administrator to view the structure and username of the credential but not alter sensitive information such as the password. The view privilege also allows the grantee to use the credential for running jobs, patching and other system maintenance operations within EM 12c. Job Credentials The job system uses the credential subsystem to retrieve the credentials required to submit a job on the targets. When submitting a job, options to use preferred credentials, named credentials or create new credentials are displayed. Monitoring Credentials The management agent uses monitoring credentials to monitor certain types of targets. Monitoring credentials can also potentially be used by management applications to connect directly to the target from the OMS.

Oracle Access Manager (OAM) SSO is the Oracle Fusion Middleware single
sign-on solution utilizing the Enterprise Directory Identity Stores.

Repository-based authentication is the default authentication option

and uses a username and password. An EM 12c administrator is also a repository DB user and, as such, may use the database password control settings via password profiles to enforce password complexity, password life time and number of failed logon attempts allowed. and centralized user identity management across the enterprise.

Oracle Application Server SSO-based authentication provides strengthened Enterprise User Security-based authentication (EUS) option enables
the creation and storage of enterprise users and roles for the Oracle database in an LDAP-compliant directory server. EUS helps centralize the administration of users and roles across multiple databases. If the managed databases are also configured with EUS, drilling down into to amanaged database allows EM 12c to directly connect to the database asthe EM 12c administrator without displaying a login page.

continued on page 10
3rd Qtr 2012 Page 9

Oracle Enterprise Manager 12c Infrastructure and Operational Security continued from page 9

Preferred Credentials Preferred credentials are used to simplify access to managed targets and set on a per user basis. With preferred credentials set, users can access a target without being prompted to log into the target. SSH Key-based Host Authentication Solaris Secure Shell (SSH) allows data to be encrypted over the network and can be used to protect against attacks such as IP spoofing, IP source routing and DNS spoofing. The agent acts as a Java SSH client and connects to the host using the username/password provided in the credential. To create SSH authentication keys, you use the SSH-keygen utility available on *NIX systems. This application provides different options to create keys with different strengths; for example, RSA keys for SSH protocol version 1 and RSA or DSA keys for use by SSH protocol version 2. SSH keys can also be used for passwordless authentication to a host. This requires proper configuration of the host and source system in order to bypass the password. Pluggable Authentication Modules (PAM) Support for Hosts Pluggable authentication modules (PAM) are used to integrate multiple low-level authentication schemes into a high-level application programming interface (API), allowing programs that rely on authentication to be written independently of the underlying authentication scheme. The use of PAM can leverage other authentication mechanisms such as LDAP, RADIUS and Kerberos. If host authentication is configured over PAM, the management agent needs to be configured accordingly to enable PAM authentication. Refer to MOS note 422073.1(6) for deployment details: Note: The local password file (usually /etc/passwd) will be checked and used first. This should be synchronized with the LDAP password if it is being used. Ifthis fails, the management agent will switch to the external authentication module. The EM 12.1 agent follows this same behavior unless the host administrator has created a PAM service called emagent. If the emagent PAM service exists, then only PAM authentication will be attempted. Sudo and PowerBroker Support EM 12c preferred credentials allows the use of two types of privilege delegation tools: Sudo and PowerBroker. Use the EM CLI or the manage privilege delegation settings page to set/edit privilege delegation settings for a host. The Sudo command allows a permitted user to execute a command as the super user or another user, as specified in the Sudoers file. If the invoking user is root or if the target user is the same as the invoking user, no password is required. Otherwise, Sudo requires that users authenticate themselves with their password. Once a user has been authenticated, a timestamp is updated and the user may then use Sudo without a password for a short period of time (five minutes unless overridden in the Sudo configuration file). EM 12c authenticates the user using Sudo and executes the script as sudo. BeyondTrust Powerbroker enables UNIX system administrators to specify the circumstances under which other people may run certain programs such as root or other important accounts. It can access existing programs, as well as its own set of utilities that execute common system administration tasks. Users can work from within a restricted shell or editor to access certain programs or files as root. See the PowerBroker documentation for detailed setup and configuration information. Authorization Authorization is the process of verifying that a user is permitted to do what he/ she is trying to do. Although that sounds simple enough, granting the proper

privileges for administrators is critical to maintaining system security as well as regulatory compliance. Many factors must be considered when assigning privileges (e.g., ensuring a separation of duties and implementing a policy of granting least privilege). It may require significant work to create and manage fine-grained access control across a large organization. Oracle EM 12c attempts to ease this burden by providing predefined roles to manage a variety of resource and target types. Please refer to the Oracle Enterprise Manager Cloud Control Administrators Guide 12c (1) for a detailed list of these roles and their functions. It is important to remember that roles within EM 12c are used to manage the privileges for the tool and not the privileges on the targets managed by the tool. For example, EM 12c default roles include entries such as EM_ALL_DESIGNER that have privileges to design operational entities such as monitoring templates; EM_ALL_OPERATOR has privileges to manage EM operations and EM_ALL_VIEWER can only view EM operations. EM 12c roles can be created to provide tool privileges such as view to a group of targets. To access or drill into the targets managed by EM 12c, the administrator would need authentication and authorization on each database target. EM 12c provides the ability to ease this administration through managed credentials or by using EUS as previously outlined in the authenticating to targets section. Privilege Propagating Groups Another feature available to assist the EM 12c administrator responsible for assigning and maintaining privileges is privilege propagating groups. Privilege propagating groups allow administrators to grant administrator privileges across a group of targets. For example, granting operator privilege on a group to an administrator grants the operator privilege on its member targets, both for current members and for members that will be added in the future. Privilege propagating groups can contain individual targets or other privilege propagating groups. Privileges on the group can be granted to a user or a role. For example, suppose a large privilege propagating group is created and granted a privilege to a role; this group is then granted to administrators. If new targets are later added to the privilege propagating group, then the administrators receive the privileges on the target automatically. Additionally, when a new administrator is hired, the role is granted to the new administrator, who receives all the privileges on the targets automatically. Audit controls Auditing is the ability to show who performed what action and when. Audit trail records can serve many purposes, such as nonrepudiation of events, regulatory compliance and discovery of security breaches. EM 12c has a robust set of auditing capabilities, as do the managed targets. For a complete record of actions, you will need to do auditing in both the EM and the managed targets. Careful consideration must be given to exactly what truly needs to be audited, because the creation of audit trail records will have an impact on the performance and capacity of the system. Also, if too many audit records are captured, audit review can become excessively time-consuming. Audit trails need to be secured in a manner that makes them tamper resistant from a nefarious user attempting to cover his or her tracks. They should also be purged or archived after they have been attested and no longer required per any regulatory mandates. By default, EM 12c creates an audit trail for all activities associated with credentials, including creating, editing, accessing and deleting them. The audited information includes current username, credential name, operation performed and operation status. The audit log contains information about the credential owner, action initiator, credential name, username, target name

Page 10 3rd Qtr 2012

and job name, along with the date and time of the operation. Sensitive fields, such as password or private keys, are never logged. To limit the amount of data stored in the repository, the audit data must be externalized or archived at regular intervals. The archived audit data is stored in an XML file stored in ODL format. For a complete list of audited operations, please see the Oracle Enterprise Manager Cloud Control Administrators Guide 12c (1).

Security Tip From the IOUG SELECT Editors

Summary Enterprise Manager Cloud Control (EM 12c) is system management software that can be deployed in various configurations to manage a wide variety of infrastructure components and in a variety of business environments. Many options exist to secure the EM 12c infrastructure and operational control to mitigate the risks associated with the business environment it is deployed to manage. Careful planning should be done before implementing these security controls so that the effectiveness and productivity gains afforded by the tool are not unnecessarily circumvented. References
1. Oracle Enterprise Manager Cloud Control Administrators Guide 12c Release 1 (12.1.0.1): http://docs.oracle.com/cd/E24628_01/index.htm. 2. Center of Internet Security (CIS) benchmarks http://benchmarks.cisecurity.org/ en-us/?route=default. 3. Oracle Fusion Middleware Securing a Production Environment for Oracle WebLogic Server http://docs.oracle.com/cd/E21764_01/web.1111/e13705/practices.htm 4. Oracle Database Security Checklist 5. http://www.oracle.com/technetwork/database/security/twp-security-checklistdatabase-1-132870.pdf 6. My Oracle Support note 422073.1 https://support.oracle.com/CSP/main/article?cm d=show&type=NOT&doctype=BULLETIN&id=422073.1. 7. Oracle Database Advanced Security Administrators Guide 11g Release 2 http://docs.oracle.com/cd/E11882_01/network.112/e10746/toc.htm#BEGIN 8. Wp-em12c-security-best-practicesv2-149383.pdf http://www.oracle.com/technetwork/ oem/frmwrk-infra-496656.html 9. Oracle Database Net Services Reference 11g Release 2 (11.2) 10. http://docs.oracle.com/cd/E11882_01/network.112/e10835/lsnrctl.htm#i553605 11. Information Security. In Wikipedia. Retrieved March 24, 2012, from http://en.wikipedia.org/wiki/Information_security 12. Pluggable authentication module. In Wikipedia. Retrieved March 24, 2012, from http://en.wikipedia.org/wiki/Pluggable_authentication_module 13. Password Strength. In Wikipedia. Retrieved March 24, 2012, from http://en.wikipedia.org/wiki/Password_strength 14. Transport Layer Security. In Wikipedia. Retrieved March 24, 2012, from http://en.wikipedia.org/wiki/Transport_Layer_Security 15. Oracle Configuration Manager Security Overview Release 10.3.0 Part Number E12881-01 http://docs.oracle.com/cd/E12482_01/apirefs.103/e12881/security.htm

A Reminder About the Vulnerability Commonly Known as the TNS Listener Poison Attack Oracle Security Alert for CVE-2012-1675
The following list includes specific suggestions for the TNS Listener Poison Attack vulnerability plus others that will help prevent TNS protocol-related attacks. 1. Standard network isolation for databases and listeners a. Limiting access by network segmentation b. Firewall 2. Limiting network access using net services a. Works to defeat several different types of TNS protocol attacks b. TCP.VALIDNODE_CHECKING=YES sqlnet.ora TCP.INVITED_NODE= (CLIENTS, NODES) sqlnet.ora c. INBOUND_CONNECT_TIMEOUT _listenername listener.ora d. SQLNET.INBOUND_CONNECT_TIMEOUT sqlnet.ora e. SQLNET.EXPIRE_TIME sqlnet.ora f. ADMIN_RESTRICTIONS_LISTENER_LISTENERNAME= ON -listener.ora 3. Named listener logs, auditing specific TNS errors a. LOG_FILE_LISTENER_LISTENERNAME= namedlogfile -listener.ora b. TNS-01169, TNS-01189, TNS-01190, TNS-12508, TNS-12546, TNS-00516, TNS-00525, TNS-00540-TNS-00560, TNS-00584, TNS-00583 4. Limiting access by database parameters a. Malformed TNS network packets b. SEC_PROTOCOL_ERROR_TRACE_ACTION c. SEC_PROTOCOL_ERROR_FURTHER_ACTION d. SEC_MAX_FAILED_LOGIN_ATTEMPTS combat brute-force type ofattacks 5. Secure transport for both RAC and single instance a. COST- class of secure transports b. 10.2.0.3 + RDBMS versions c. RAC- 1340831.1 (requires SSL/TLS; usage added to license) d. Single instance 1453883.1 e. Requires a patch 12880299 6. Turn off dynamic registration a. DYNAMIC_REGISTRATION=OFF in listener.ora b. Disables certain features such as RAC, DataGuard or PL/SQL Gateway for APEX. Please test in a nonproduction environment, making a single change at a time. SQLNET Tracing, as well as auditing the protection level, can help in troubleshooting, too.

About the Author

Janet Wakeley, CISSP,, works for GE Healthcare as the data management security leader with more than 14 years of experience on various Oracle technologies and more than half that time focused on security-related concerns.

3rd Qtr 2012 Page 11

TRANSFORMED
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. Copyright 2012 EMC Corporation. All rights reserved. 120020

MISSION-CRITICAL APPS

Resolving Child Cursor Issues Resulting in Mutex Waits

consuming more CPU cycles. This procedure, which every SQL statement has to go through at least once in its life, is usually referred to as a hard parse. Parent and Child Cursor Now that you know how expensive this hard parse process is, Oracle started looking for a way to avoid this effort as much as possible. Oracle introduced the concept of cursor sharing where databases store all parsed SQL in the library cache section of shared pool. Now it is named a cursor. We should distinguish between a parent cursor, which contains the static metadata like the SQL full text, and a child cursor, which has all the dynamic variables. Note: If we have the same SQL but different metadata, there will be more children for one parent.

By Martin Klier Edited by Arup Nanda

ave you ever seen an issue as a result of waiting for mutex waits in Oracle Database? If you run an average application, I am fairly sure you have seen this more than once, even more so in later versions of Oracle, and especially in Oracle Database 11g Release 2.

Shared Pool Check The library cache is where the parsed SQL statements reside. It is organized as a hash tree, based on the hash value of the literal SQL text coming from the DB user interface. As soon as an SQL successfully goes through a syntax and semantic check, the parsing process performs the shared pool check, which checks inside this hash tree structure if an SQL with the same text is already stored. If it isnt, the hard parse described above will take place, and, afterward, a new parent/child cursor pair is placed in the library cache.

In this article, you will learn some elementary database concepts such as parsing and serialization and corresponding common wait events, as well as examples and demonstrations of commonly occurring issues and their explanation. Finally, you will see links to further readings such as MOS notes and publications by other authors. Lets start with some basic concepts about parsing and cursors. When an SQL statement is sent for execution, it has to be validated, interpreted and stored in a form that Oracle Database can understand before it can be executed. This process is known as parsing. As part of the parsing process, the database performs several steps: a syntax check, semantic check, cost-based optimization of the query execution plan, row source generation, etc. After parsing is complete, the parsed code is stored in a special area of the memory called shared pool. In this article, I will explain some shared pool issues; however, covering the entire process in detail is beyond the scope of this article. Figure 1: Enterprise Manager Components and Communication Links Soft Parse If a matching cursor definition is already found in the library cache, the database happily reuses the existing cursor and execution plan to save time. This process is what performance analysts call a soft parse and what we nearly always want to have compared to a hard parse. Bind Variables To be able to benefit from soft parsing, all of us have to issue very similar but not identical SQL statements (think of SELECT something FROM table WHERE anything=13). This is pretty much next to impossible. Therefore, Oracle introduced the concept of bind variables. The bind variables are usually visible with a :(colon) before their name in the SQL source code and can be supplied with actual values by the application at run time. Here is an example of an SQL with a bind variable:
SELECT something FROM table WHERE anything=:X;

Cursor Sharing Hard Parse The SQL executed in the database is usually from an application, which connects to the database using JDBC, OCI or other methods. After ensuring that syntax (SQL grammar) and semantics (such as objects and data types) are valid, the parsing process optimizes the access path between various database objects. In relatively simple cases, the process boils down to just the choice between index and full table access. In more complex situations, the access path is a combination of more than one table and their corresponding indexes. The database performs a sort of brute-force approach to this problem and classifies all results by their artificial cost factor. Despite the fact that this cost-based optimizer employs a number of shortcuts in generating access paths, you can appreciate the fact that creating an execution plan may be a lengthy and expensive process. In this context, expensive means

The application supplies the value of the variable X at the run time. Since the SQL statement is identical regardless of the value of the variable, the hash value remains the same, avoiding unnecessary hard parsing. Disadvantage of Cursor Sharing In dynamic environments, cursor sharing with bind variables has an unfortunate downside. Suppose Oracle is parsing our example SQL anything=:X (=42) for
continued on page 14
3rd Qtr 2012 Page 13

Resolving Child Cursor Issues Resulting in Mutex Waits continued from page 13

the first time. Assume in our table there are millions of entries with 42 in the anything column. So the selectivity of this predicate is almost zero. The parsing process will usually generate an optimal access as a full table scan. The execution plan and the corresponding cursor are stored in shared pool, indexed by the hash value of the SQL statement. Now suppose the example SQL is parsed a second time but with a different value of :X, 43. Assume there is only one row that matches this predicate. However, since the hash value of the SQL statement is the same, Oracle will decide to soft parse and reuse the plan generated earlier. The database will perform a full table scan even for this single row selectivity, increasing unnecessary buffer gets and consequently more CPU cycles and longer elapsed time. A better plan would have been index scan; but the soft parse will not come to the conclusion since it will find the parsed statement in the library cache and will assume the original execution plan.

Figure 3: Adaptive Cursor Sharing II With this harder soft parsing, Oracle not only peeks into the bind variable at initial parse time, but every time a known SQL comes into the parsing process. This process and the comparison with the child cursors already in the library cache need resources, of course.

Adaptive Cursor Sharing (11g and Above) Clearly, there is a problem. This is why Oracle introduced adaptive cursor sharing in Oracle Database 11g Release 1 to reduce the negative impact of this selectivity problem. Prior to 11g, new child cursors were created for lots of reasons but never due to different selectivity based on bind variable values.
Cardinality Feedback Lets digress a little to introduce a slightly different concept here: cardinality feedback. Introduced in Oracle Database 11g, it can identify a parent cursor if the size of the result set does not match the expected outcome of the cost-based optimization process. If a parent cursor is marked as such and the next hard parse gives a different plan, a new child cursor is created and located under the old parent.

Cursor Creation and Invalidation Oracle documentation and various MOS notes explain 64 reasons as to why a new cursor might be created or an old one may become invalid. This article only lists a summary of the few most important ones.
Optimizer Environment If the optimizer mode changes between executions (ALL_ROWS vs. FIRST_ ROWS), it is more likely that a new execution plan has to be created. So invalidating the existing cursors here is needed to prod the optimizer to generate a new plan. Outline Mismatch It is the same for mismatched outlines. If another outline is given for the new environment where the SQL runs, an existing cursor may be incorrect, especially its execution plan. NLS and User Identity Issues If we are issuing SQL statements from varying NLS environments, different sorting or filtering may be needed. So it is not a given that all objects can be used in the same way. For instance, function-based indexes may change between different NLS parameter environments. Additionally, if we issue the same SQL statement from a different schema with the same object names by coincidence, the cursor, although similar in characters, is actually very different. The database has to create new cursors. The decision is made by the parsing process by examining the parsing schema. Cardinality Feedback We examined this topic in the last section. If Oracle decides to have another execution plan for a bind set, either a new cursor has to be created or, in case it already exists, the child cursor hosting the old plan will need to be invalidated and replaced. Bind Mismatch Bind mismatch is the most unwanted reason, as I will demonstrate in later sections. Bind mismatch means that something about the bind variables was different between executions. The list of reasons is long, but the length of the variable and datatype differences are the most common ones.

Figure 2: Adaptive Cursor Sharing I There are two modes in cardinality feedback. The parent cursor can be marked as:

bind sensitive, i.e., the optimizer wants the cursor to be re-parsed in order
to find out if theres a potentially better plan

bind aware, which occurs when theres more than one child cursor for

selectivity reasons. This cursor is paired with a bind set information to get the correct plan.

Hard, Soft and Harder Soft Parse After a parent cursor had been marked bind aware, the next parsing of this SQL cant be a true soft parse anymore. Now the optimizer has to decide which of the existing execution plan and child cursor combination is best for the bind selectivity given by the user. Since the entire statement is not parsed, this is not hard parse; but this is not soft parse either. Its sort of a harder soft parse examining the bind information given for each cursor and deciding which existing child/plan combination should be reused.
Page 14 3rd Qtr 2012

Mutexes Serialization

Mutex Contention

Figure 4: Linked List - Initial State While using CPU resources for ordinary memory operations is bad, accessing or changing existing cursors in memory with absolute integrity is worse. The process of changing the cursor needs serialization; no process wants its own memory structure manipulated by another while reading or writing it. For example, lets take a look at the classic singly linked list, shown in Figure4. Figure 7: Mutex Contention Nevertheless, mutex contention still is possible and not as uncommon as many may think. As always, contention comes from parallel execution being serialized: Two processes are applying for the same resource. One holds it; the other one wants it. In the simplified example shown in Figure 7, session 1 uses the library cache object/cursor in question. It also holds the corresponding mutex, which is responsible serialization structure. Session 2 wants to use the same library cache object cursor in an incompatible way and, therefore, contends for the mutex protecting it. Repeatedly asking for the status of a mutex is known as spinning where a tiny piece of memory is queried over and over again. As you can imagine, this intensely consumes CPU. To allow the OS scheduler and other multitasking mechanisms to use this CPU as well, there is a maximum spin count defined. It is the database initialization parameter _spin_count. When this count is up, the process sleeps for this mutex. After a while, it comes back and spins again, until the mutex is freed by session 1. Waits Especially since 10gR2 and 11g, the Oracle Wait Interface is a powerful source of information for performance analysis and shows information coming from Oracles code instrumentation. The Oracle kernel code itself reports those waits, and so we get a quite good impression what is (not) going on at the moment. As explained in last section, contentions cause active spinning, loading the CPU with calls without doing real work. As soon as the mutexes sleep for a while, Oracle records this a so-called wait event. This article focuses on cursor-related mutex wait events, so we will discuss the top five of them in greater detail. Cursor: Mutex X The X wait event tells us that an exclusive lock on a cursor-related mutex is happening. Processes are holding this mutex, or are contending for it, as soon as they do or want to:

Figure 5: Linked List - After Removal Node i-1 is linked to node i, which is connected to node i+1. If we want to remove nodes i and i+1, we have to unlink them from their predecessor, as shown in Figure 5. But the problem is that if we do the removal simultaneously, even though we are successful in deleting node i by relinking node i-1 to node i+1, we abort the deletion of node i+1 in the same operation, as shown in Figure 6.

Figure 6: Linked List - Result Although the other thread unlinked node i+1 successfully, its not deleted because we immediately linked node i-1 to it; so it failed. The only way to do this properly is to do it one step at a time; this is what serialization means. Mutex is a mechanism to accomplish serialization. All Linked List graphics are coming from Wikipedia, Author: KMVisor. The Structure A mutex is an abbreviation for mutual exclusion, and it means a fine-grained serialization structure. Basically, it works the same way as a latch does, but its lighter weight, often directly hardware-supported and, thus, fast. As a result, its suitable for a widespread use. Starting in 10gR2, Oracle replaces lots of latches by mutexes. Being faster and having a smaller footprint in memory, a mutex provides a double improvement on performance. Speed is only one part of the puzzle; those little structures need to be shared, too, so that false contentions are reduced. Latches often have been shared between many cells of a structure; mutexes are usually dedicated exclusively to one logical piece. In our case this means that each parent and child cursor has its own mutex, and we do not have to contend for a latch thats simultaneously used by many cursors.

Insert a new child cursor under a parent Do bind peeking for it (=capture SQL bind data) Modify cursor-related statistics

Figure 8: LC Object Mutexes

continued on page 16
3rd Qtr 2012 Page 15

Resolving Child Cursor Issues Resulting in Mutex Waits continued from page 15

Cursor: Mutex S The S wait event tells us that a shared lock on a cursor-related mutex is happening. Processes are holding this mutex, or are contending for it, as soon as they do or want to do a change in the reference count of the mutex itself. This count indicates the number of processes interested in this mutex. The basic meaning is a new guy is interested, and, thus, spinning/waiting. Cursor: PIN X The pin X wait indicates sleeps on an exclusive mutex protection against deleting a structure while its manipulated. Processes will hold it or contend for it when they are or want to be creating or altering the cursor. Cursor: PIN S The pin S wait indicates sleeps on a sharable mutex protection against deleting a structure while its used. Processes will hold it, or contend for it, when they are or want to be in use of a cursor. Nobody wants to have their cursor definition deleted while an SQL statement is still running. Cursor: PIN S Wait on X The pin S wait on X wait indicates a contention for a shared lock on the pin mutex of a cursor while its still held in exclusive mode by another process. This usually happens when one session tries to execute the cursor but another one is actually altering it. An example is concurrent DDL on an object used by the cursor we want to reuse.

create or replace PROCEDURE LOAD7(LOOPS IN NUMBER DEFAULT 1000) is begin for i in 1..LOOPS loop execute immediate select 1 from dual where 1=2; end loop; end; / SHOW ERRORS; /

And this is the corresponding bash script, used to start 20 threads with 10,000,000 executions each:
#!/bin/bash LP=10000000 sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm

Issues, Quick Fixes and Long-Term Solutions Several issues are known for cursor-related mutex waits. This article will elaborate only the most common ones, including some unnecessary Oracle bug-related incidents.
Simple: Cursor: PIN S Problem Figure 9 shows an Oracle Enterprise Manager screen, particularly the top activity/SQL detail. Its a SELECT statement. The base (green) field is CPU time; the top (blue) is cursor: pin S waiting (which we we know now is mutex sleeping). The red line above the graph is the number of CPUs 12 of them in this case. A part of the green corresponds to the mutex of our SELECT statement spinning. But, of Figure 9: Waits Cursor: pin S course, theres other work to do as well (buffer gets, for example). Please note: the cursor: pin S wait probably wont be a problem here but, nevertheless, it happens. Test Case Here is the first test case. Its a SELECT statement executed within a tight PL/SQL loop. Twenty of these loops are running concurrently, so we are soft parsing (at best, hoping for no hard parses) the same cursor hundreds of times per second. This is package LOAD7, used for loop in the example:

<<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec

load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);& load7($LP);&

You will observe the sessions waiting for cursor: pin S. This is an indicator for a kind of hot spot object in the library cache. The most convenient way to find out if and how much we are waiting for cursor: pin S is the wait interface. Other avenues, such as the top activity screen in Enterprise Manager, AWR or Statspack reports, also work. Solution The basic solution may sound trivial: Dont parse this statement so often. In rare cases, frequently viewed web pages or fast running parallel batch runs issuing the same SQL statements may see this, too. In those cases, the only solution for too many cursor: pin S waits is to break up the SQL hash value or SQL_ID of your statements, e.g., by adding a process or server name to a comment within its text:
select /* WebServer4 */ something from table;

Please use this little hack wisely. Too many different SQL statements in the database will end up with hard parsing all the time.

Page 16 3rd Qtr 2012

Complex: Cursor: Mutex S/X Problem 1 Figure 10 shows a screenshot from Oracle Enterprise Managers top activity/SQL details. We see the results from an UPDATE statement. From the bottom to top of the graph: the green portion is CPU time; the second (yellow) is a generic library cache exclusive serialization; the third (orange) is cursor: mutex X wait (the one we are interested Figure 10: Waits: Cursor Mutex S/X in); and, on the top, in dark green, we can see a little bit of the cursor: mutex S wait. The red line toward the upper portion shows the number of CPUs 12 of them in this case. From the diagram you can see that a part of the CPU time is spent in mutex spinning. Yet, there is other work to do as well (such as working on buffers, etc.). We see a massive concurrency wait here; many processes are contending for exclusive usage of cursors. It means they are frequently trying to change the cursors something that deserves closer investigation. The system is clearly heavily overloaded; there are 20 sessions are actively waiting for something, but there are only 12 CPUs. Test Case The second test case is also an UPDATE statement, which is manipulating 10 columns of a row with as many data type change and variants as possible. The inner loop of package LOAD6 ensures that we are going through all variants coming from a bit structure. If the bit representing the current column is 1, we are setting it with the NUMBER data type, and if its 0, the column will be set with VARCHAR2.
/* Code idea and working prototype by Dietmar Br at ATU, thank you very much! */ create or replace PROCEDURE LOAD6(p_ChildLimit IN NUMBER DEFAULT 8, LOOPS IN NUMBER DEFAULT 1000, p_RowNumber IN NUMBER DEFAULT 1) IS TYPE t_arrayc IS VARRAY (11) OF VARCHAR2 (2048); TYPE t_arrayn IS VARRAY (11) OF NUMBER; TYPE t_bindstrings IS VARRAY (11) OF VARCHAR2 (256); v_bindsc t_arrayc := t_arrayc ( 1,1,1,1,1,1,1,1,1,1,1); v_bindsn t_arrayn := t_arrayn ( 1,1,1,1,1,1,1,1,1,1,1); v_BindStrings t_BindStrings := t_BindStrings ( :ZERO,:ONE,:TWO,:THREE,:FOUR,:FIVE, :SIX,:SEVEN,:EIGHT,:NINE,:TEN); n NUMBER; c NUMBER; d NUMBER; v_BinStr VARCHAR2 (15); v_BitStr VARCHAR2 (15); BEGIN dbms_output.enable; c := DBMS_SQL.open_cursor; DBMS_SQL.parse ( c,

UPDATE /*+LOAD6-1*/ TEN_NUMBERS SET ONE = :ONE, TWO = :TWO, THREE = :THREE, FOUR = :FOUR, FIVE = :FIVE, SIX = :SIX, SEVEN = :SEVEN, EIGHT = :EIGHT, NINE = :NINE, TEN = :TEN WHERE ZERO = :ZERO, DBMS_SQL.native); DBMS_SQL.bind_variable (c, :ZERO, p_RowNumber); FOR i IN 1 .. LOOPS LOOP v_BinStr := TRIM (TO_CHAR (dbms_numsystem.dec2bin (round(dbms_random.value(1,p_ChildLimit),0)), 00000000000)); FOR i IN 2 .. 11 LOOP v_BitStr := SUBSTR (v_BinStr, i, 1); IF (v_BitStr = 1) THEN DBMS_SQL.bind_variable (c, v_BindStrings (i), v_bindsn (i)); ELSE DBMS_SQL.bind_variable (c, v_BindStrings (i), v_bindsc (i)); END IF; END LOOP; n := DBMS_SQL.execute (c); END LOOP; DBMS_SQL.close_Cursor (c); COMMIT; END; / /* Code idea and working prototype by Dietmar Br at ATU, thank you very much! */

And here is the corresponding bash script, used to start 20 threads with 64 variants ($CL), 400,000 executions each ($LP), as well as making sure not to hit the same row/block to avoid buffer busy waits (third parameter).
#!/bin/bash CL=64 LP=400000 sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm sqlplus -s klm/klm

<<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec <<<exec

load6($CL,$LP,1000); & load6($CL,$LP,2000); & load6($CL,$LP,3000); & load6($CL,$LP,4000); & load6($CL,$LP,5000); & load6($CL,$LP,6000); & load6($CL,$LP,7000); & load6($CL,$LP,8000); & load6($CL,$LP,9000); & load6($CL,$LP,10000); & load6($CL,$LP,11000); & load6($CL,$LP,12000); & load6($CL,$LP,13000); & load6($CL,$LP,14000); & load6($CL,$LP,15000); & load6($CL,$LP,16000); & load6($CL,$LP,17000); & load6($CL,$LP,18000); & load6($CL,$LP,19000); & load6($CL,$LP,20000); &

After both scripts are run, we would be continuously invalidating cursors due to the reason of BIND_MISMATCH and generating lots of child cursors. Its expensive for the parsing process to look for the matching one, change it and insert a new child cursor all these times. The test case with 64 variants here has been enough to saturate a 12 core machine completely.
continued on page 18
3rd Qtr 2012 Page 17

Resolving Child Cursor Issues Resulting in Mutex Waits continued from page 17

Real-Life Example The earlier test case was meant to be an example only rare to occur in real life. Here is a case reproducing a real-life Java application:

The application connects with an Oracle JDBC driver, and the second
bind variable (#2) in the cursor is manipulated by a setter method: setNUMBER(2). Obviously, the result is the bind data type NUMBER.

Trace 2: Bind#2 >> oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=01 csi=873 siz=0 off=168 kxsbbbfp=110977db8 bln=22 avl=02 flg=01 value=99

Next time, when we need to set bind #2 to NULL, one way with JDBC is to
manipulate the cursor with method setNULL(2). But this sets the bind variables data type to VARCHAR2, which leads to cursor invalidation by BIND_MISMATCH and creation of a new child cursor.

One or a few occurrences probably wont be visible, but setting around 30 columns to numeric or null this way results in theoretically 2^30 child cursors for this single SQL statement. The effects on library cache and its serialization structures will be devastating in this case. In reality, you wont even have to wait for that high number. As soon as more than 100 child cursors have been in the library cache, the cursor: mutex s/x waits will become more pronounced. In this case, the limitations of the server dictated an upper limit of around 500 cursors, beyond which saturation prevented further activity on the machine. The system in question was an 8 CPU AIX server, parsing around 100 of those hash-identical UPDATEs per second. Diagnosis Initially, this issue may be discovered by measuring the high CPU load or thebad response time. Usually, as a next step, the DBA checks the wait interface with the tool of their choice and discovers the mutex wait events discussed earlier. You can easily find out why cursors became invalidated through a simple query on v$sql_shared_cursor, but you have to be careful. Sometimes the information provided is not very reliable (see the section on pitfalls later). Ihighly recommend using the new 11.2.0.2 event cursortrace level 16 explained in My Oracle Support document ID 296377.1. It generates a large, but clearly readable, trace file. Inside the trace file, search for your SQL_ID and you can see the reasons of the performance issue. MOS engineers will typically ask for this cursor trace dump file immediately when you open an SR. Now that you have determined the reason for the cursor invalidation, the next challenge is to find out which bind variables are the culprits. Unfortunately, the application developers will need that piece of information to be able to find the position in their code. But good old 10046 level 12 trace dumps out the information we need. Tracing all sessions in such a load situation will be impractical, of course. Fortunately, Oracle 11g introduced another quite cool trace event named sql_trace for a particular SQL_ID, making it easier to trace a specific SQL statement:
SQL> alter system set events sql_trace [sql:<SQL_ID>];

Quick Fix How do you fix this? Emergency often makes quick and dirty measures look good. One such desperate intervention for an issue with too many cursors in the system might be as simple as flushing them out frequently. I wont recommend flushing the entire shared pool, though, because this will very likely cause other issues. But since version 10.2.0.4 plus a patch, or at least with 10.2.0.5, we have the option to purge out one particular parent cursor by using the dbms_shared_pool.purge function. This is a statement to create a valid dbms_shared_pool.purge command for a known SQL_ID, because we need the address and hash_value of the cursor:
select exec sys.dbms_shared_pool.purge (||address||,||hash_value||, C); from v$sqlarea where sql_id=4dwguxwc8h6gx;

If you need to purge frequently, another suggestion is to create a dbms_ scheduler job for kicking out the parent cursor on an appropriate interval (one minute worked great in our real-life example we are talking about). Nevertheless, the database may switch to other problems now; cursor: pin S wait on X or high hard parsing rates may be side effects of this workaround. Long-Term Solution Of course, we need a solution for the basic problem: The setter method setNULL(2) we used to manipulate our cursor shifted the data type of a bind variable formerly being NUMERIC to now being VARCHAR2. The JDBC specification says that handing over the data type to setNULL() is optional. It defaults to VARCHAR2 if we dont explicitly set the datatype. The real solution for bind variable #2 to be and to stay on NUMERIC/Integer is to call this statement:
setNULL(2, java.sql.Types.INTEGER)

This probably will need to involve development and cause software changes, but it will be a future-proof fix. Problem 2 A Similar Case with BIND_LENGTH_UPGRADEABLE Here is another case, similar to the above case with numeric binds but different because there is another known problem with variant CHAR bind variables. The database does adjust the length of CHAR bind buffers to preset sizes. There are four steps: 32, 128, 2,000 and 4,000 bytes. So if we execute a statement with a bind value of 10 bytes, the buffer will be 32 bytes. If we re-execute the same statement with a value of 52 bytes in the bind variable, the previously created child cursor cannot be reused and will be recreated with a bind buffer of 128 bytes. The system view v$sql_shared_cursor indicates this invalidated child cursor as BIND_LENGTH_UPGRADEABLE. Having a few of those invalidated child cursors may not harm the system. They will age out regularly as every invalidated cursor does, but if we have high load or concurrency, the aging out will not be as fast and Oracle may create many of these child cursors. Why? Lets go back to numeric issue example described earlier. Combinations matter; the more CHAR bind variables we have in a single statement, the more the number of combinations possible for one

Lots of trace files will be generated, but you will be able to compare same bind positions with same bind positions. As you can see in the trace excerpt taken during the real-world case from last section above, the oacdty is different on this bind position (#2), the data type has changed.
Trace 1: Bind#2 >> oacdty=01 mxl=32(04) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000010 frm=01 csi=873 siz=0 off=168 kxsbbbfp=1118e1cd8 bln=32 avl=00 flg=01

Page 18 3rd Qtr 2012

cursor. Since it is possible to have up to four possible definitions per bind buffer, you may end up with 4^n child cursors where n is the number of different combinations. The effect is similar to whats described in the numeric issue case. We will continuously invalidate the cursors with the reason BIND_LENGTH_UPGRADEABLE and consequently generate lots of child cursors. For the parsing operation, checking for the matching cursor, changing it and inserting a new child cursor all the time is extremely expensive on CPU cycles. Solution As a really quick-and-dirty solution, the emergency measure using the dbms_shared_pool.purge function as described above works here as well. Keep in mind this may end up in trading parsing CPU load for mutex waits, so use it with caution. A long-term solution is preferred, but that is the complicated part because nothing can be said in general terms. You should decide on a solution based on your specific situation.

realizing the negative side effects. Since we implicitly use the new features of the parser and other built-in functions, applications have to make sure that they are stepping around the limitations. Oracle developers cant determine what third-party developers all over the world are doing. All they can do is to merely come up with their own specifications when publishing a new functionality. DBAs need to make sure they are cognizant of those specifications. The bugs described above are a result of developing an incredibly complex product, and we will have to live with such issues. For Resolving Child Cursor Issues Resulting in Mutex Waits, there are some suggestions for casualties of this topic:

Check how the application handles its bind variables. Avoid

BIND_MISMATCH at any cost.

Reduce the number of cursor versions (= child cursors) to below 100;

every single child will increase load and effort.

Heavy: Oracle Internal Pitfalls

Behavior Change From 10g To 11g Oracle 10g theoretically was also vulnerable to cursor mutex issues because it already had similar child cursor structures, mutexes and cursor invalidation mechanisms. But the changes (or, improvements, as Oracle calls them) around cardinality feedback and adaptive cursor sharing seem to have made it worse for parsing issues. If we take a look at the harder soft parse concept described above, thats somehow understandable. Oracle followed several approaches to improve the speed of cursor handling. One of them slimming down the kernel code works to the disadvantage of applications producing too many child cursors. You may wonder why a system on 10g did not produce tons of child cursors while the new 11g database did. Here is the official answer from Oracle: Its important to note that cursor obsoletion code was removed in version 11. That means we no longer obsolete a parent cursor when it reaches 1024 child cursors. As a workaround, Oracle recently published Enhancement Patch 10187168, which introduces the hidden parameter _cursor_obsolete_threshold. Only use underscore parameters under guidance of Oracle Support. In 11.2.0.3, this patch is included and _cursor_obsolete_threshold defaults to 1024 again. Bugs Like 10157392 and 12939876 Both bugs are related and based upon each other. They are about a memory leak in 11g code that causes a creeping increment of child cursor numbers, especially when the shared pool is under stress. In the end, we have all cursor: mutex wait events discussed above without doing anything wrong. The usual diagnosis via v$sql_shared_cursor does not show any reason, and cursortrace indicates this bug by dumping lots of (such as in thousands) bind mismatch (3) cursor versions. The bugs are fixed in 12.1 and backported to 11.2.0.3. No fix is included in any 11.2.0.2 patch bundle. Bug 9591812 This bug is about wrong wait events in 11.2 (cursor: mutex S instead of cursor: mutex X). The somewhat official MOS workaround says: Be cautious when interpreting S mode mutex / pin waits. So we are officially told not to believe or at least to suspect anything what we see. The bug is fixed in 12.1.

If in trouble, look up MOS for Oracle bugs matching your issue. Upgrade your 11g to 11.2.0.3 or higher.
Additional Readings
My Oracle Support (MOS) Document IDs 1356828.1

1377998.1 296377.1
References
Pder, Tanel Year 2011 Presentation: Oracle Latch and Mutex Contention Troubleshooting http://www.slideshare.net/tanelp/oracle-latch-and-mutex-contention-troubleshooting Shallahamer, Craig Book: Oracle Performance Firefighting (ISBN 978-0-9841023-0-3) http://resources.orapub.com/Oracle_Performance_Firefighting_Book_p/ff_book.htm Nikolaev, Andrey Year 2011 Blog entries: Mutex waits. Part 1 + 2 http://andreynikolaev.wordpress.com/2011/07/09/mutex-waits-part-1%E2%80%9Ccursor-pin-s%E2%80%9D-in-oracle-10-2-11-1-invisible-and-aggressive/ and http://andreynikolaev.wordpress.com/2011/10/25/mutex-waits-part-ii-cursor-pin-s-inoracle-11-2-_mutex_wait_scheme0-steps-out-of-shadow/

About the Author

Summary Cursor mutex wait events are a problem related to the internal workings of Oracle Database and applications using the power of the RDBMS without

Being in the IT business as a specialist for Linux, Martin Klier (martin.klier@klug-is.de) has been involved in the administration of Oracle Databases for about 10 years and has worked on a senior level for more than three years. The integration of large Oracle-based high-availability solutions (MAA) with RAC and Data Guard have been the first challenges. In the last five years, he largely moved into performance analysis and tuning of complex systems with dynamically changing load profiles. Klier frequently blogs about current issues and C their solutions at www.usn-it.de.

3rd Qtr 2012 Page 19

Application Development with Oracle Advanced Queuing

There are three general categories of messaging:

Single consumer (i.e., point-to-point) a message is dequeued by a

single consumer themessage

Multicast the producer effectively names designated consumers for Broadcast consumers may dynamically gain access to a message
queue by subscribing A robust messaging systems supports a wide variety of features in addition to those describe above. These include:

By Jeffrey Jacobs Edited by John Kanagaraj

Error handling Timeouts and expirations Enqueuing/dequeuing of messages as a group Dequeueing messages by criteria other than FIFO, including, but not limited to: Enqueue time Priority Contents of messages Reliability Propagation pushing messages to destinations Other queues Other databases Other messaging systems (e.g., JMS, middleware, gateways) Retention of messages and history of actions Nonrepudiation Logging Performance evaluation Warehousing Wide range of message content data types (i.e., payload), including: Text XML BLOB, LOB, CLOB Structured records

his paper is intended to provide the reader with an understanding of the basic features and capabilities of Oracle Advanced Queuing (AQ) for consideration in application development. It does not cover all the features and capabilities of Oracle AQ. The reader should consult the Oracle Documentation (Oracle Streams Advanced Queuing Users Guide and Oracle Database PL/SQL Packages and Types Reference) prior to beginning application development. Oracle AQ provided PL/SQL, OCI, JMS and SOAP APIs. While they all offer virtually identical functionality, this paper refers only to the PL/SQL packages, DBMS_AQ and DBMS_AQADM.

What Is Messaging? Messaging is the ability to send a message containing data from one application/process to another application/process. It is a widely used technique in distributed systems, particularly high volume OLTP systems. Unlike client server applications, which are typically synchronous, messaging is typically asynchronous (i.e., the sender, referred to as the producer, is not blocked waiting for a reply from the recipient(s), referred to as consumer(s)). Oracle Advance Queuing (AQ) does not support synchronous messaging.
Messaging has many uses and advantages. It allows applications and systems to communicate and cooperate in an API independent manner. An order-entry system may send a message containing order information to a fulfillment system without requiring access to internal APIs. The same message may also simultaneously be routed to an inventory management system, a customer support application, an email acknowledgment application, etc. Messages are placed into queues, called enqueuing. The enqueuing applications are called the producers. There is typically no restriction on the number of producers for a given queue. The application data portion of the message is referred to as the payload. Messages are read and removed from the queue by dequeuing the message. Applications dequeuing messages are referred to as consumers.

Notification to consumers of message availability Guaranteed delivery High performance

Oracle AQ provides all of this functionality. In addition, Oracle AQ also provides the ability to browse messages without dequeuing.

Queue Types Oracle AQ provides the three types of messaging described above via two basic types of queues: single-consumer queues and multiconsumer queues. A multiconsumer queue may provide both multicast and broadcast capabilities. All queues allow any application with appropriate permissions to enqueue messages. In a single consumer queue, a given message is dequeued by only one consumer, after which it is removed from the queue. However, multiple consumers may dequeue from the queue (e.g., multiple instance of an application, such a multiple instances of a fulfillment application processing messages from a single

Page 20 3rd Qtr 2012

order queue). Single-consumer queues have the simplest underlying structure and, when used appropriately, typically offer the highest performance. Queues need to be started after creation via START_QUEUE. Queues can be stopped via STOP_QUEUE. Both procedures allow control of enqueuing and dequeuing separately. For multiconsumer queues, the determination as to whether a message is broadcast or multicast is made at the time the message is enqueued; it is not a property of the queue itself.

sort_list determines the order in which messages are normally

dequeued. It applies to all queues and governs the generation of the underlying queries. This can be overridden by certain dequeuing options, but it cannot be changed after creation. The default is enqueue time, which is effectively FIFO. this type.

multiple_consumers TRUE or FALSE. All queues in the AQ table are of message_grouping NONE or TRANSACTIONAL.
If TRANSACTIONAL, all messages enqueued in one transaction may be treated as a group when dequeuing. See Transaction Protection below:

Message States A message may be in one of the following states:

READY message is available to be dequeued WAITING availability for dequeuing is delayed EXPIRED message has timed out and been moved to exception queue PROCESSED message has been consumed by all consumers

comment a description of the AQ table that will be stored in the

datadictionary.

primary_instance primary owner of the queue table service (RAC);

seeRAC Considerations below.

Buffered Messaging Buffered messaging is a lightweight, nonpersistent form of messaging, which can be specified at the time of enqueuing. It is generally only memory resident and does not support many of the features that are available for persistent messaging. In particular, buffered messages do not support:

secondary_instance secondary owner of the queue table service (RAC);

see RAC Considerations below.

compatible lowest database version compatibility (only 10gR2 and later

are covered in this paper).

secure TRUE for secure queues (not covered in this paper).

RAC Considerations Each AQ table effectively creates a service. AQ table structures are typically hot tables with a great potential for hot blocks. To avoid performance issues caused by cache contention, the services should be pinned to a single node (i.e., node affinity). primary_instance specifies the preferred instance on which the service will run. secondary_instance specifies the preferred instance if primary instance is not available. If neither instance is available, a random instance is selected. Creating Queues Queues are created via:
DBMS_AQADM.CREATE_QUEUE ( queue_name IN VARCHAR2, queue_table IN VARCHAR2, queue_type IN BINARY_INTEGER DEFAULT NORMAL_QUEUE, max_retries IN NUMBER DEFAULT NULL, retry_delay IN NUMBER DEFAULT 0, retention_time IN NUMBER DEFAULT 0, dependency_tracking IN BOOLEAN DEFAULT FALSE, comment IN VARCHAR2 DEFAULT NULL, auto_commit IN BOOLEAN DEFAULT TRUE);

Grouping Retention Guaranteed delivery Array dequeuing

Advanced Queuing (AQ) Tables An AQ table is an abstract object type that may be implemented by one or more underlying tables, indexes and index organized tables depending on whether the AQ table supports single or multiconsumer queues. An AQ table typically holds one or more queues, which can be created and destroyed dynamically.
Multiconsumer AQ tables typically require more management and overhead. AQ tables are created by:
DBMS_AQADM.CREATE_QUEUE_TABLE( queue_table IN VARCHAR2, queue_payload_type IN VARCHAR2, [storage_clause IN VARCHAR2 DEFAULT NULL,] sort_list IN VARCHAR2 DEFAULT NULL, multiple_consumers IN BOOLEAN DEFAULT FALSE, message_grouping IN BINARY_INTEGER DEFAULT NONE, comment IN VARCHAR2 DEFAULT NULL, primary_instance IN BINARY_INTEGER DEFAULT 0, secondary_instance IN BINARY_INTEGER DEFAULT 0, compatible IN VARCHAR2 DEFAULT NULL, secure IN BOOLEAN DEFAULT FALSE);

The parameters are described below:

The relevant parameters are described below:

queue_table AQ table name queue_payload_type payload type storage_clause any valid storage clause. Tablespace should always be

queue_name the name of the queue queue_table the name of the AQ table holding queue queue_type NORMAL_QUEUE or EXCEPTION_QUEUE max_retries the maximum number of dequeue retries before moving to exception queue; see Transaction Protection below

retry_delay after a failure (usually ROLLBACK), the number of seconds

before message will be available for dequeuing again afterdequeuing

specified. Oracle recommends using ASSM. If ASSM is not used, INITRANS and PCTFREE may be set if needed for extremely high transaction queues; this has not been necessary in the authors experience.

retention_time the time the message remains in the queue table

continued on page 22
3rd Qtr 2012 Page 21

Application Development with Oracle Advanced Queuing continued from page 21

dependency_tracking not currently implemented comment Queue documentation, which is kept in the data dictionary auto_commit deprecated
Enqueue Options and Features There is a wide range of options for enqueuing messages. These options include, but are not limited to:

BUFFERED the message is only maintained in memory and may be

lost in the event of system failure or database shutdown

sequence_deviation deprecated as of 10.2 relative_msg_id effectively deprecated sequence_deviation effectively deprecated

DBMS_AQ.MESSAGE_PROPERTIES_T The DBMS_AQ.MESSAGE_PROPERTIES_T record is used for both enqueuing and dequeuing operations.
TYPE message_properties_t IS RECORD ( priority BINARY_INTEGER NOT NULL DEFAULT 1, delay BINARY_INTEGER NOT NULL DEFAULT NO_DELAY, expiration BINARY_INTEGER NOT NULL DEFAULT NEVER, correlation VARCHAR2(128) DEFAULT NULL, attempts BINARY_INTEGER, recipient_list AQ$_RECIPIENT_LIST_T, exception_queue VARCHAR2(61) DEFAULT NULL, enqueue_time DATE, state BINARY_INTEGER, sender_id SYS.AQ$_AGENT DEFAULT NULL, original_msgid RAW(16) DEFAULT NULL, signature aq$_sig_prop DEFAULT NULL, transaction_group VARCHAR2(30) DEFAULT NULL, user_property SYS.ANYDATA DEFAULT NULL delivery_mode PLS_INTEGER NOT NULL DEFAULT DBMS_AQ.PERSISTENT);

Enqueuing a single message Enqueuing an array of messages (PL/SQL or OCI) Message grouping, which treats all messages enqueued in a single
transaction as a group

Sender identification Time specification and scheduling of message delivery Correlation identifier, which allows multiple messages queued with a user
defined identifier to be dequeued together.

Enqueuing Message The following PL/SQL API is used to enqueue messages:

DBMS_AQ.ENQUEUE( queue_name IN VARCHAR2, enqueue_options IN enqueue_options_t, message_properties IN message_properties_t, payload IN type_name, msgid OUT RAW);

The relevant enqueue attributes are:

priority the priority of the message. This is only relevant if the sorting
method specified for the table includes the priority.

queue_name the name of the queue in which the message is to be enqueued payload the type definition of the payload, typically, but not limited to, a
PL/SQL abstract type

delay specifies number of seconds before a message is available for

dequeuing. The default is 0 (NO_DELAY).

expiration the number of seconds a message is available for dequeuing

(after delay). If the message is not dequeued by all subscribers, it will be moved to the exception queue with a status of EXPIRED. This is necessary for multiconsumer queues, as not all subscribers may be able to dequeue the message. The default is the constant NEVER.

msg_id the unique identifier of the message

DBMS_AQ.ENQUEUE_OPTIONS_T The DBMS_AQ.ENQUEUE_OPTIONS_T record contains the options for enqueuing the message as described below:
TYPE SYS.ENQUEUE_OPTIONS_T IS RECORD ( visibility BINARY_INTEGER DEFAULT ON_COMMIT, relative_msgid RAW(16) DEFAULT NULL, sequence_deviation BINARY_INTEGER DEFAULT NULL, transformation VARCHAR2(61) DEFAULT NULL, delivery_mode PLS_INTEGER NOT NULL DEFAULT PERSISTENT);

delivery_mode DBMS_AQ.BUFFERED or DBMS_AQ.PERSISTENT

determines if the message is buffered or persistent. The default is persistent.

correlation the ID used for dequeuing by correlation ID. This is a

producer-supplied value, which allows a logical grouping of messages. Unlike a transaction group, the messages need not be enqueued in a single transaction or by the same producer.

The attributes are:

visibility ON_COMMIT the message is enqueued as part of the transaction

(i.e.,enqueuing the message is completed by COMMIT) IMMEDIATE the message is enqueued immediately in an autonomoustransaction. before enqueuing (not covered in this paper)

Dequeuing Features Oracle AQ provides very high performance and functionality. Key features include:

transformation Specifies a transformation function to be performed delivery_mode PERSISTENT the message is stored in the queue table
Page 22 3rd Qtr 2012

Concurrent dequeues Multiple dequeue methods and options Array dequeue Message navigation Waiting for messages Retries with delays

 Transaction protection Exception queues

DBMS_AQ.DEQUEUE The PL/SQL API is:
DBMS_AQ.DEQUEUE( queue_name IN VARCHAR2, dequeue_options IN dequeue_options_t, message_properties OUT message_properties_t, payload OUT type_name, msgid OUT RAW);

Number the wait time in seconds. Process is blocked while waiting.

The next message is dequeued on wake up. Note: Oracle AQ also offers the ability for a process to listen on multiple queues, but the functionality is outside the scope of this paper.

Dequeue Methods There are several methods for dequeuing messages. The default is to dequeue individual messages based on the sort order specified when the AQ table wascreated.
Note: the most efficient navigation method for dequeuing based on the sort order is to use NEXT_MESSAGE without FIRST_MESSAGE. FIRST_MESSAGE always performs a query. However, if NEXT_MESSAGE is used without FIRST_MESSAGE, it will only perform one SELECT in the session; subsequent calls are simple fetches. Other methods are:

Note that message_properties_t is used for both enqueue and dequeue operations.

DEQUEUE_OPTIONS_T
TYPE DEQUEUE_OPTIONS_T IS RECORD ( consumer_name VARCHAR2(30) DEFAULT NULL, dequeue_mode BINARY_INTEGER DEFAULT REMOVE, navigation BINARY_INTEGER DEFAULT NEXT_MESSAGE, visibility BINARY_INTEGER DEFAULT ON_COMMIT, wait BINARY_INTEGER DEFAULT FOREVER, msgid RAW(16) DEFAULT NULL, correlation VARCHAR2(128) DEFAULT NULL, deq_condition VARCHAR2(4000) DEFAULT NULL, signature aq$_sig_prop DEFAULT NULL, transformation VARCHAR2(61) DEFAULT NULL, delivery_mode PLS_INTEGER DEFAULT PERSISTENT);

Correlation ID dequeue series of message based on correlation

asfollows:

Get Correlation ID by dequeuing using FIRST_MESSAGE. Dequeue

additional messages via NEXT_MESSAGE using the value of correlation until no more messages remain. The specification for correlation may use pattern matching (%,_).

This method typically requires the addition of an index and generation of statistics to force the underlying queries to use the index on the correlation column.

The DBMS.AQ.DEQUEUE_OPTIONS_T specifies the dequeuing options as described below:

Transaction group similar to correlation but uses transaction_group

consumer_name the name of the subscriber. dequeue_mode. Modes include: REMOVE (with data) this is the typical dequeuing method. The

set by producer. Should use array dequeuing but may use same loop as Correlation ID above, specifying the transaction_group. Pattern matching may also be used. of payload object elements or other columns. See documentation for more details about specifying columns and payload elements. Note that using the method supersedes all other methods.

deq_condition similar to the SQL WHERE clause, it accesses contents

message may remain in the queue table for history based on retention period, but it not eligible for future dequeuing (unless via msg_id). REMOVE_NODATA no data is returned, but the message is removed from queue. This may be used for selective cleanup. BROWSE reads the message data but does not actually dequeue the message. The message remains available for future processing (unless dequeued by another process). Browsing may not be repeatable, and, assuch, there are numerous gotchas to be aware of.

msgid dequeue a single message by system-assigned RAW value. This

typically requires browsing the queue(s) and is usually used for cleanup and corrections.

Dequeue visibility Messages may be dequeued in the following modes:

navigation there are two methods for navigation when dequeuing. FIRST_MESSAGE This creates a snapshot (effectively a cursor); note
that this only retrieves messages that were enqueued at the time of the dequeue call. NEXT_MESSAGE If FIRST_MESSAGE was used, this retrieves the next message in the snapshot. See Default Dequeuing below. message. The options are: FOREVER waits forever, which is the default. Typically used for high frequency queues. Note that this blocks the process. NO_WAIT dont wait for next message. Typically used for deferred or batch operations, which are initiated by jobs scheduled at regular intervals.

IMMEDIATE Messages are removed from the queue in an autonomous

transaction. If the application does not have retry capabilities, this will typically offer better performance and scalability

ON_COMMIT (transaction protection) Messages are removed from the

wait if no messages are available, the consumer may wait for the next

queue on COMMIT of the transaction. The dequeue operation is treated in the same manner as an INSERT/UPDATE/DELETE. If the transaction fails either due to ROLLBACK, system failure or shutdown the retry count is incremented. If the retry count is exceeded, the message is moved to the exception queue; otherwise, it remains in the original queue. Note that a system failure or shutdown may not increment the retry count. If retry_delay was specified when the queue was created, the message will not be available for dequeuing for the specified number of seconds.
continued on page 24
3rd Qtr 2012 Page 23

Application Development with Oracle Advanced Queuing continued from page 23

Message Expiration If expiration is specified in message_properties_t.expiration, all consumers must dequeue the message before expiration time. Otherwise, the message is moved to the exception queue. It is generally a good practice to specify expiration for multiconsumer queues, as not all consumers may be active, which would result in the message remaining in the queue indefinitely. Exception Queues Each AQ table has at least one exception queue that contains messages that have expired or exceeded retry count from all of the other queues. Messages in an exception queue may be dequeued once by only one consumer for reprocessing. Exception queues should be monitored and periodically emptied either for reprocessing or simply free space. Propagation Messages may be pushed to other queues via propagation. Those queues typically, but not always, exist in another database or an external messaging system; the latter is beyond the scope of this paper. Propagation may also be to queues in the same database. The messages are ultimately processed by consumers of the destination queue(s); propagated messages are considered process upon completion of propagation. Propagation may push messages to multiple queues in multiple targets (i.e., fan out). Messages may also be propagated from multiple sources into a single queue. The destination queue may be single or multiconsumer but must be of the same payload type. Propagation is performed by scheduled jobs. A propagation window is a period of time in which propagation can occur (i.e., effectively scheduling the job).
There are two basic modes for propagation between databases:

Other APIs to manage propagation are:

ALTER_PROPAGATION_SCHEDULE DISABLE_PROPAGATION_SCHEDULE ENABLE_PROPAGATION_SCHEDULE SCHEDULE_PROPAGATION VERIFY_QUEUE_TYPES

AQ Table Structures A multiconsumer AQ table has seven underlying tables, both heap and index organized. The main table with message data for all queues has the same name as specified in CREATE_QUEUE_TABLE, e.g. ORDERS_QUEUETABLE. Other tables have names beginning with AQ$, e.g. AQ$_ORDERS_QUEUETABLE_H. A single consumer AQ table creates a single table with main table name; the index structure may vary. Performance Tips for Dequeuing Using certain features, such as Correlation ID or Transaction Grouping, may require additional indexes on the main table. To change the behavior of the queries used by AQ, statistics need to be gathered, as AQ tables are exempt from automatic statistics gathering. However, generating appropriate statistics in a production environment can be problematic due to the volatile nature of queues; stopping the queues to allow messages to build up in order to gather statistics is probably not acceptable to the DBAs. Statistics can either be created manually, or, in a development or QA environment, messages can be enqueued without dequeuing. The statistics can then be imported into production for the table. Its also a good idea to lock the statistics just to be safe. Query to be Tuned Finding the underlying dequeuing query for tuning is not immediately obvious. Look in appropriate V$ or GV$ views or AWR report for the following pattern:
SELECT /*+ FIRST_ROWS(1) */ tab.ROWID, tab.user_data FROM <queue_table_name> -- the name of the main queue table WHERE q_name = :1 AND (state = :2 and ORDER BY q_name, FOR UPDATE SKIP LOCKED;

Queue to dblink effectively deprecated Queue to queue the target queues are specified
The API to schedule propagation is:
DBMS_AQADM.SCHEDULE_PROPAGATION ( queue_name IN VARCHAR2, destination IN VARCHAR2 DEFAULT NULL, start_time IN DATE DEFAULT SYSDATE, duration IN NUMBER DEFAULT NULL, next_time IN VARCHAR2 DEFAULT NULL, latency IN NUMBER DEFAULT 60, destination_queue IN VARCHAR2 DEFAULT NULL);

The parameters are:

queue_name the name of the queue to be propagated destination destination dblinks start_time the start time for the propagation (i.e., the time when the
job will first be schedule)

FOR UPDATED SKIP LOCKED is the secret sauce for AQs performance. It performs SELECT FOR UPDATE only on rows that are not currently locked. It also apparently only locks rows when they are fetched, but this has been difficult to confirm. This is not a documented or user supported feature.
About the Author

duration how long propagation lasts in seconds. NULL means the

propagation lasts forever (or until stopped or altered).

next_time a calendar expression (as used by DBMS_SCHEDULER) for

the next propagation window

latency if no messages, how many seconds to wait until checking the

queue for message to be propagated. Zero results in propagation as soon as a message is available.

Jeffrey Jacobs is a data architect at PayPal and a member of the board of directors of IOUGs Oracle Exadata SIG. Jacobs presented Application Development with Oracle Advance Queuing at COLLABORATE 2012 and C was also recognized as an Oracle ACE in February 2012. He can be reached at www.jeffreyjacobs.com.

Page 24 3rd Qtr 2012

Time for Change: Migrate your Non-RAC Database to RAC

Benefits of Using Oracle RAC Scalability: Oracle Real Application Cluster provides scalability for all your enterprise business applications. Oracle provides a wide array of tools and techniques for scaling, and you can use these tools to ensure seamless growth while minimizing the investment in hardware resources. You can allow RAC database to grow seamlessly from a small system to a big multinational enterprise applications.
High Availability: Oracle RAC provides reliability, recoverability, continuous operations and error detection features for a highly available data management. If an instance fails in Oracle RAC database, the cluster detects the problems immediately and recovery will start automatically. The remaining instances in the server remain active and open for uses.

By Satishbabu Gunukula Edited by Gary Gordhamer

Maintenance: In Oracle RAC, most of the database maintenance operations can be performed without downtime, and many other maintenance tasks can be done in a rolling fashion (rolling patch, rolling upgrade) so application downtime is minimized.

racle Real Application Cluster was introduced in Oracle 9i and it is now proven technology used by thousands of customers in every industry for every type of application. Oracle RAC provides the options for scaling applications beyond the capabilities of a single server. This allows customers to take advantage of lower cost commodity hardware to reduce their total cost of ownership and provide a scalable computing environment that supports their application workload. Learn about different approaches to convert a non-RAC database to RAC database.
The Oracle RAC eliminates the ability to remove the server as a single point of failure in any database application environment. It enables the Oracle Database to run mainstream business applications of all kinds of server pools, including products such as SAP, PeopleSoft and Oracle Applications, which can be either DSS, OLTP or mixed workload.

How to Convert Single Instance to RAC Oracle provides the following methods to convert a single instance database to RAC. You can choose any method based upon your convenience:
1. DBCA 2. RCONFIG (from 10g R2) 3. Enterprise Manager 4. Manual (Using RMAN)

Figure 2 We are assuming that the source single-instance database and the target RAC database are the same release of Oracle and are running on same platform. If the versions are different, use Oracle Database Upgrade Assistant (DBUA) to upgrade the database and use one of the methods to convert a single instance database to RAC. Before you convert your single-instance database to RAC, make sure that you back up your single-instance database and configure your shared storage so it is accessible from all nodes in the RAC environment cluster. The diagram change describes the difference between single-instance storage and RAC storage.

Figure 1
continued on page 26
3rd Qtr 2012 Page 25

Time for Change: Migrate your Non-RAC database to RAC continued from page 25

DBCA automates the database configuration during single instance to RAC conversion and minimizes the manual intervention of tasks. DBCA performs the below tasks automatically:

Creates undo and redo logs required for RAC database Configures control file attributes Creates the initialization parameter entries required for a cluster-enabled
environment

Configures Oracle Net Services and cluster resource Configures Oracle RAC database to manage using Oracle Enterprise Manager Configures the RAC database to manage by using SRVCTL utility
Figure 3: RAC Database Storage Principles We have different shared storage options for Oracle RAC, but Automatic Storage Management(ASM) is the recommended method. Using ASM has many benefits, such as online storage migration, easy management and automatic I/O tuning. Please find the available shared storage options in Oracle RAC Before you convert single instance to RAC, you need to create an image of single instance using DBCA and then perform the actual conversion process using DBCA. Refer to the Oracle documentation via the link below for information on how to convert single instance to RAC using DBCA. http://docs.oracle.com/cd/E11882_01/install.112/e18069/ cvrt2rac.htm#BABBBDDB

ASM Automatic Storage Management RAW Raw devices (only supported for older releases) CFS Vendor cluster file system (e.g., OCFS, QFS, GXFs) NFS Network file system (certified NAS)

Refer to the Oracle documentation via the link below for information on Automatic Storage Management (ASM). http://docs.oracle.com/cd/E11882_01/server.112/e18951/toc.htm

RCONFIG RCONFIG is a noninteractive command line utility that comes default on Oracle 10g and above versions. This utility uses the information in ConvertToRAC.xml in order to convert single instance to RAC. You need to prepare input XML with convert options according to your environment.
This utility has the ability to migrate non-ASM to ASM, and it internally uses RMAN to copy the data from non-ASM to ASM. This utility uses the RMAN default configuration when copying data and does not allow you to modify the configuration parameters. As a result, RMAN uses one channel to copy the database, which means a longer downtime. If your database is very large, it might not be a good option for you. If you see any errors in the middle of the conversion when using this method, you need to perform cleanup on the converted instance and reinitialize the conversion process. In some cases, we have noticed the conversion fails to add resources to CRS. At this stage, instead of redoing the whole process, you can add the resources to CRS manually using SRVCTL. In one of our tests, we are able to convert a 400GB database in around five hours using RCONFIG. We are able to convert the same database in around two-anda-half hours using RMAN parallelism. When using RMAN, there are many parameters to optimize the performance and more room for improvement. Below are the common issues when using RCONFIG to convert single instance to RAC. 1. When using ASM as shared storage, makes sure to mention listed ASM information in ConvertToRAC.xml file. Otherwise, RCONFIG will not be able to detect ASM storage and conversion will fail. 2. For the actual conversion process, you must set the value of convert verify to YES in ConvertToRAC.xml file or conversion will fail. 3. If you dont mention the valid disk group names in ConvertToRAC.xml file, the data file will be created in the $ORACLE_HOME/dbs local file system, and you will receive an error message that the data file did not share across all nodes in the cluster. For example: Make sure you specify the valid diskgroup names in ConvertToRAC.xml file

Automatic Storage Management (ASM) Best Practices We have tested several best practices in our test tab, and we have seen very good results. Several of these results are mentioned in Oracle documentation/ publications.

Install ASM on a separate Oracle Home for high availability and manageability. Use multipathing over multiple HBAs to provide I/O load balancing and
failover capability.

Remove ASM dependency on VIP. In case VIP fails, the ASM instances
remains operational.

For better performance, use ASM diskgroups with four or more disks, and
make sure these disks span several back-end disk adapters.

Use EXTERNAL redundancy if you are doing the mirroring at storage level. If you are using hardware RAID, make sure the LUN stripe size is as close
to 1MB as possible

Always use diskgroups with disks that are similar in size and performance.
A diskgroup with large number of disks provides a wide distribution of data extents. This allows greater concurrency for I/O and reduces the occurrences of hotspots.

Lets discuss the options that you have to convert single instance to RAC.

DBCA Most DBAs are familiar with DBCA, but before using DBCA for conversion, make sure that system hardware and operating system are supported.

Page 26 3rd Qtr 2012

<n:SharedStorage type=ASM> <n:TargetDatabaseArea>+DGROUP1</n:TargetDatabaseArea> <n: TargetFlashRecoveryArea>+DGOUP2</n:TargetFlashRecoveryArea> </n:SharedStorage> 4. The host names in ConvertToRAC.xml must be in the same case as actual host names ($hostname) or the conversion process will fail with the following error: The cluster is not configured or is not running on node. 5. The SourceDBHome, TargetDBHome must be specified in ConvertToRAC.xml file or the conversion will fail with the following error message: oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: Cannot get SQLEngine . The error is: java.io.IOException: /home/oracle/product/10.2.0/db_1/bin/sqlplus: not found Refer to the Oracle documentation via the link below to convert single instance to RAC using RCONFIG. http://docs.oracle.com/cd/E11882_01/install.112/e18069/cvrt2rac. htm#BABGGEGJ

This method includes two steps: 1. Duplicate single instance database into RAC system as single instance 2. Convert single instance to RAC In this method, you will be using the RMAN DUPLICATE DATABASE command to duplicate your single-instance database to RAC as single instance. Once the database is duplicated on RAC node, you will need to add redo and undo for second instance, enable all cluster-related parameters and register with CRS. Below are the common issues when using RMAN to convert single instance toRAC: 1. The user must create the password file for auxiliary database or you will receive the following error: RMAN-04006: error from auxiliary database: ORA-01031: insufficient privileges 2. Make sure you add the auxiliary database TNS entry on target database (non-RAC) or you will receive the following error: RMAN-04006: error from auxiliary database: ORA-12154: TNS:could not resolve the connect identifier specified

Enterprise Manager Oracle Enterprise Manager is a GUI-based database management tool with which you can perform most of your database management and admin tasks. If you want to convert your single instance to RAC without much user intervention, then this method is best for you.
Refer to the Oracle documentation via the link below to convert single instance to RAC using Enterprise Manager. http://docs.oracle.com/cd/E11882_01/install.112/e18069/cvrt2rac. htm#BABBAAEH

How to Back Up Oracle RAC Databases RMAN is a command-line and Enterprise Manager-based tool and is included with the Oracle server. Using RMAN for backup is the Oracle-preferred method for efficiently backing and recovering Oracle RAC databases. RMAN automatically detects the bock corruptions, and you can also validate backups without actually restoring.
RAC database backups are the same as single-instance backups; you can run the backup and restore from any node in the cluster. The best practice is to use a shared location accessible by all nodes. If you enable the auto backup of control file features, RMAN automatically restores the SPFILE if required for instance recovery. It can also recover the database even if the current control file, catalog and server parameter file arelost. Here are few compelling reasons to adopt RMAN for database backups:

Manual (Using RMAN) I would recommend using RMAN if your database is a very large database (VLDB) and you want to have full control over the conversion process. It has many features and parameters, and you can optimize the performance based upon your requirement. You can use RMAN to convert a database running on file system to ASM.
If you have more than one disk group and you want to place data files across different disk groups, then you can use the SET NEWNAME command for each data file in the duplication script and RMAN will restore the data file accordingly.

Trouble-free backup and recovery Corrupt block detection Archive log validation and management Block media recovery (BMR) Easily integrates with media managers Backup and restore optimization Backup and restore validation Downtime-free backups Incremental backups Extensive reporting

Refer to the Oracle documentation via the link below to configure Recovery Manager for backup. http://docs.oracle.com/cd/E11882_01/rac.112/e16795/rman.htm

Figure 4
continued on page 28
3rd Qtr 2012 Page 27

Time for Change: Migrate your Non-RAC database to RAC continued from page 27

Best Practices, Tips and Hints Below are several best practices, tips and hints that have been tested at our lab with very good results. A number of them are mentioned in Oracle documentation/publications.

Prior to 11g R2, it is recommended to run the listener from the ASM home.
From 11g R2 on, the recommended method is to run the listener from the grid infrastructure home (by default). furnished with both VIP and public host names.

To avoid name resolution issues, ensure that the HOSTS files and DNS are If using NFS disks for RAC, make sure that correct mount options are used.
The mount options are detailed in Document 359515.1 for each platform. simplifies client connectivity and eliminates the need to modify database connect strings when the cluster grows and/or shrinks. interface cards (NICs) and private interconnect NICs on each cluster node to avoid public and private networks from being the single point of failure. Dedicated redundant switches are highly recommended for the private interconnect (crossover cables are not supported). block corruption) can occur if LUNs larger than 2TB are presented to an ASM diskgroup. As a result of the fix, ORA-15099 will be raised if a disk larger than 2TB is specified. This is irrespective of the presence of asmlib.

In a RAC environment, avoid using a shared Oracle Home. If you install

software in a shared Oracle Home, then you cannot perform a rolling upgrade of patches/sets. Also, binaries will have local dependencies and single point of failure. and online storage migration (adding, removing disks). versions of Oracle Database.

It is highly recommended to utilize SCAN for your 11.2 databases, as it

Choose ASM as shared storage for automatic I/O tuning, easy management Oracle ASM supports both older (must be at 10.1.0.3 or higher) and newer Before you set up a RAC environment, check with the disk vendor that the
number of nodes, OS version, CRS version, RAC version, network fabric and patches are certified, as some storage/san vendors may require special certification for a certain number of nodes. first before moving the changes to production. Also, the test environment should mirror the production environment as closely as possible.

Oracle highly recommends configuring a redundant set of public network

Do not add more than a 2TB size disk to a disk group. ORA-15196 (ASM

It is highly recommended to test patches and upgrades in a test environment It is strongly advised that a production RAC instance does not share a node
with a DEV, QA, test or TRAINING instance. These extra instances can often introduce unexpected performance changes into a production environment. groups for redundancy.

Always multiplex control files, redo logs and archive logs across disk The VIPs and SCAN VIPs must be on the same subnet as the public interface. Prior to 11g, it is recommended to store vote and OCR on RAW devices.
For additional information, see the Understanding SCAN VIP whitepaper. From 11g R2, ASM is the recommended method for storing VOTE and OCR. It is recommended to maintain no more than two ASM disk groups for storing VOTE and OCR.

Conclusion Oracle Real Application Cluster allows the enterprise application to grow its business in any direction by providing protection from hardware, software failures and ensures continuous data access. Oracle RAC is designed for scalability and high availability. Many customers implemented Oracle RAC for their mission-critical applications. If you havent migrated your singleinstance database to RAC yet, now is the time. Please refer to the links below for additional information:

Please check the link for Oracle Database High availability Solutions

USERS GROUP CALENDAR

For the most updated calendar, please visit www.ioug.org SEPTEMBER 2012
September 30October 4 Oracle OpenWorld Moscone Center, San Francisco Event URL: http://www.oracle.com/ openworld/index.html

DECEMBER 2012
December 12 NYOUG Special Winter GeneralMeeting New York Oracle Users Group 8:30 p.m. 5 p.m. New Yorker Hotel 481 Eighth Ave. (34th St.) New York, NY 10001 Event URL: http://www.nyoug.org

http://www.oracle.com/technetwork/database/features/availability/ index.html Having a step-by-step plan for your RAC project implementation is invaluable. Please check OTN Article for sample project outline: http://www.oracle. com/technetwork/articles/haskins-rac-project-guide-099429.html Oracle Clusterware portion can always be installed in a rolling upgrade fashion: HOWTO: Note: 338706.1 For detailed information on Single Client Access Name (SCAN), refer Document 887522.1 Understanding SCAN VIP white paper. For recommended Patches check Metalink Note 756671.1 Refer to the Oracle documentation link for Oracle Real Application Cluster Installation: http://docs.oracle.com/cd/E11882_01/install.112/ e24660/toc.htm

OCTOBER 2012
October 17-18 East Coast Oracle Users Conference Sheraton Imperial Hotel & ConventionCenter Raleigh/Durham, NC Event URL: www.eastcoastoracle.org

About the Author

APRIL 2013
April 7-11 COLLABORATE 13 Denver, Colo. Event URL: http://collaborate13. ioug.org/p/cm/ld/fid=112

Satishbabu Gunukula has more than 13 years of experience in Oracle and SQL Server Database technologies. He specializes in high-availability solutions (Oracle RAC, Data Guard, Grid Control, SQL Server Cluster) and has implemented many business-critical systems for Fortune 500 and 1000 companies. Gunukula is an Oracle-certified DBA in 8i/9i/10g and an Oracle-certified expert in 10g RAC. He was awarded the Oracle ACE by Oracle Corporation. He shares knowledge through his websites http://www.oracleracexpert.com C and http://www.sqlserver-expert.com.

Page 28 3rd Qtr 2012

Use Case Comparison of Oracle 11g on Solid State Disks (PartTwo)

Note that the conventional writes are asynchronous by default. The process does not have to wait for writes to be written to disk. Our interest is to see Oracle write metrics. For this purpose, we can deploy direct path load to measure I/O speeds. The direct path write Oracle metrics are provided when a process that is writing buffers directly from PGA and the process waits on this event for the write call to complete. This is in contrast to the conventional path, whereby DBWR writes the dirty buffers from the buffer cache (not the process itself). The normal conventional path insert would be free to use the existing free space, whereas a direct path insert will not; it will always allocate new space above the high water mark. Operations that could perform direct-path writes include, among others, direct-path inserts. For example, any of the following will use a direct-path load:
create table as select ... or insert /*+ append */ into table select ....

By Mich Talebzadeh Edited by April Sims

Using direct path eliminates much of the Oracle Database overhead by formatting Oracle data blocks and writing the data blocks directly to the database files. We described a number of waits in the first part of this article. We will briefly touch on Oracle waits associated with direct-path writes. Direct-Path Write During direct-path operations, the data is asynchronously written to the database files. At some stage, the session needs to make sure that all outstanding asynchronous I/Os have been completed to disk. This can also happen if, during a direct write, the session is out of free slots and has to wait to store the outstanding load requests (a load request could consist of multiple I/Os). This will show as direct-path write wait event. Additional Notes on Redo Log Files For the purpose of these tests, the redo log files were created identically on HDD and SSD disks. For tests on HDD, the redo logs were on HDD. For SDD, the redo logs were dropped and created on SSD. Scenario 1: SSD versus HDD for Bulk Inserts Expected outcome: SSD will fare reasonably better compared to HDD but not that much faster. The initial seek time for SSD will be faster, but the rest of the writes will be sequential extent after extent and will not be much different from what HDD does. In summary:

n the first part of this two-part article on solid state disks (SSD), we covered the advantages that SSD offers over the magnetic spinning hard disks (HDD) when it comes to read-intensive activities. In this second part, we will consider the use case of solid state disks for Oracle 11g tablespace file systems covering write-intensive activities.
Introduction As we learned in Use Case Comparison of Oracle 11g on Solid State Disks (Part One), which was published in the Q2 2012 issue of SELECT Journal, database performance is ultimately constrained by the need to read and write data from a persistent storage source. For the introduction to SSDs, the rationale in using SSDs, the environment setup and the approach we will be using, please refer to back to part one. The purpose of this second part is to examine the use case for solid state disks in terms of the performance of Oracle in write-intensive activities. Measuring the Volume of Read and Write in Oracle Each user process in Oracle does its own read from the disks. In contrast, for conventional path, Oracle uses database writer background process (DBWR) to write changed (dirty buffers) data and to undo data to disk. The log writer process (LGWR) writes redo log data to redo log files.
With writes, the outcome is that one will not see physical writes typically with the conventional path, because they are buffered until a checkpoint happens. With a checkpoint, every dirty block in the buffer cache is written to the data files. In other words, it synchronizes the datablocks in the buffer cache with the datafiles on the disk. A checkpoint is initiated when:

The first seek time for positioning the disk on the first available space will
be much faster for SSD than HDD.

The rest of the writes will have marginal improvements with SSD over HDD.
Test Case: We base our test case on table tdash that we introduced in part one. Just to recap, table tdash is populated with 1,729,204 rows of random data based on all_objects table. The structure of this table is as follows:
CREATE TABLE TDASH ( OWNER VARCHAR2(30 BYTE) NOT NULL ENABLE, OBJECT_NAME VARCHAR2(30 BYTE) NOT NULL ENABLE, SUBOBJECT_NAME VARCHAR2(30 BYTE), OBJECT_ID NUMBER NOT NULL ENABLE, DATA_OBJECT_ID NUMBER, OBJECT_TYPE VARCHAR2(19 BYTE),

There is a redo log switch LOG_CHECKPOINT_TIMEOUT has expired LOG_CHECKPOINT_INTERVAL has been reached DBA explicitely issues a checkpoint via the alter system checkpoint command

continued on page 30
3rd Qtr 2012 Page 29

Use Case Comparison of Oracle 11g on Solid State Disks (Part Two) continued from page 29 CREATED DATE NOT NULL ENABLE, LAST_DDL_TIME DATE NOT NULL ENABLE, TIMESTAMP VARCHAR2(19 BYTE), STATUS VARCHAR2(7 BYTE), TEMPORARY VARCHAR2(1 BYTE), GENERATED VARCHAR2(1 BYTE), SECONDARY VARCHAR2(1 BYTE), NAMESPACE NUMBER NOT NULL ENABLE, EDITION_NAME VARCHAR2(30 BYTE), PADDING1 VARCHAR2(4000 BYTE), PADDING2 VARCHAR2(4000 BYTE) ); Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 93 Rows Row Source Operation ------- --------------------------------------------------0 LOAD AS SELECT (cr=3253250 pr=1123356 pw=1122708 time=0 us) 1000000 COUNT STOPKEY (cr=3243260 pr=1122898 pw=0 time=94500480 us) 1000000 TABLE ACCESS FULL TDASH (cr=3243260 pr=1122898 pw=0 time=93204184 us cost=216894 size=8100000000 card=1000000)

This table is created on an 8K database on both SSD and HDD schemas, and, in order to ensure that each record in the table is larger than a block, two columns PADDING1 VARCHAR2(4000) and PADDING2 VARCHAR2(4000) were added and populated. The table had a primary key on object_id column as follows:
ALTER TABLE tdash ADD CONSTRAINT tdash_pk PRIMARY KEY (object_id);

Misses in library cache during parse: 1 indicates that this was the first time we ran this query (after a reboot). The Row Source Operation outputs provide the number of rows processed for each operation executed on the rows and additional row source information. Here there was a full table scan of tdash table fetching a million rows. The cost to retrieve a million rows from this table was pr=1,122,898 physical blocks. It took time=94,500,480 microseconds. The consistent reads were cr=3,243,260 (logical block reads). The optimizer estimated the costing for this operation to be cost=216,894 units returning size=8,100,000,000 bytes of data with cardinality card=1,000,000 rows. COUNT STOPKEY is a count operation where the number of rows returned is limited by the ROWNUM expression in the WHERE clause. It is important to note that the execution plan has a hierarchical structure and parents I/O, cost, timing, etc., is a sum of corresponding metrics for all its children plus the metrics of the operation itself. In the above, the COUNT STOPKEY is the parent operation of TABLE ACCESS FULL TDASH. The childs statistics are rolled into the parents statistics. In the same way, LOAD AS SELECT is the parent operation of the execution plan. In this case there were 1,123,356 physical block reads. The consistent reads were 3,253,250 blocks. The important metric to note was the number of physical block writes. These were pw=1,122,708 blocks. Those 1000,000 rows were processed in one execution of the SQL statement. Toretrieve and insert those million rows it cost 0.70 seconds of CPU time and 198.99 seconds of elapsed time. There were 3,244,233 queries (logical block I/Os) that included 1,122,918 physical block reads. Also there were 1,141,845 block reads from the buffer cache in current mode. A current mode get, also called a db block get, is a retrieval of a block as it currently appears in the buffer cache. For example, if an uncommitted transaction has updated two rows in ablock, then a current mode get retrieves the block with these uncommitted rows. The database uses db block gets most frequently during modification statements, which must update only the current version of the block. In our case it was the INSERT. Note the CPU time and elapsed time differences. Further, we will need to investigate the cause of this elapsed time. The Oracle wait event section in TKPROF report shows:
Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- -----------Disk file operations I/O 3 0.00 0.00 db file sequential read 28 0.06 0.26 direct path read 4880 0.20 67.50 direct path write 27218 0.18 121.85 ********************************************************************************

Our focus is to see the time it takes to write a million rows to disk on HDD and SSD schemas. Thus, tdash table in each schema will be the source table. We will deploy create table as select for this test. The SQL code will look like the following:
ALTER SESSION SET TRACEFILE_IDENTIFIER = tester_ctas_testwrites; EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>ALL_ EXECUTIONS ); CREATE TABLE testwrites AS SELECT * FROM tdash WHERE ROWNUM <= 1000000; commit;

It is a simple and common code. It will take the first million rows from tdash table and put them in a newly created table testwrites. Note the enabling of trace and wait events:
EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>ALL_EXECUTIONS );

The TRACE output will be written to TRACE directory. It is a detailed file and large. One can use TKPROF utility to summarize the TRACE output. A simple output in this case excluding system calls will be sufficient
TKPROF <TRACE_FILRE> <OUTPUT_FILE> SYS=NO

The interest would be in getting various matrices for the bulk load operation. The results from HDD follow:
CREATE TABLE testwrites AS SELECT * FROM tdash WHERE ROWNUM <= 1000000 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.69 198.98 1122918 3244233 1141845 1000000 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ---------total 2 0.70 198.99 1122918 3244233 1141845 1000000

To retrieve those records in total, they were 4,880 direct path read physical I/O taking 67.50 seconds (note physical I/O not blocks). There were 27,218 direct path write physical I/O taking 121.85 seconds in total. The writes were in multiblocks. From the trace file we can see the details for each WAIT for direct path write. For example:

Page 30 3rd Qtr 2012

AIT #5: nam=direct path write ela= 4931 file number=11 first dba=521216 block cnt=32 obj#=84459 tim=1337078795854580

The typical elapsed time differences are certainly better for SSD compared to HDD (2,353 vs. 4,931 microseconds) but not by order of magnitude.

The above line indicates that in one physical I/O write there were cnt=32 blocks written, and it took ela= 4931 microseconds. Now we will try the same with the schema on the solid state disks. For the sake of brevity, we will not display the details here. The following summary report is from TKPROF output:
CREATE TABLE testwrites AS SELECT * FROM tdash WHERE ROWNUM <= 1000000 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.86 97.15 1122918 3244235 1138177 1000000 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ---------total 2 0.86 97.16 1122918 3244235 1138177 1000000 Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 96 Rows Row Source Operation ------- --------------------------------------------------0 LOAD AS SELECT (cr=3247332 pr=1122931 pw=1122708 time=0 us) 1000000 COUNT STOPKEY (cr=3243260 pr=1122903 pw=0 time=33636212 us) 1000000 TABLE ACCESS FULL TDASH (cr=3243260 pr=1122903 pw=0 time=32343496 us cost=216894 size=8100000000 card=1000000)

SSD Versus HDD for DML Operations Via Index Scan Up to now we were looking at the performance of query on SSD versus HDD for bulk inserts. So how does SSD fare for typically day-to-day DML operations (INSERT/UPDATE/DELETE) via index scan? What performance improvements are we going to see in these cases? We need to simulate a typical OLTP workload with frequent insert/update/deletes and commits and then look at the results and ask ourselves: Did we get more work done faster because we were using solid state disks?
To keep matters simple, we will use the newly created testwrites table that has one million rows. We will then create a primary key constraint for this table on OBJECT_ID column:
ALTER TABLE testwrites ADD CONSTRAINT testwrites_pk PRIMARY KEY (object_id);

Note that the cost of this operation on SSD in terms of disk, query, current, number of rows, table access cost in terms of consistent reads, physical block reads, costing, size and cardinality are more or less the same as what we got from HDD. The only differences are the timings. With SSD disks we have slightly higher CPU time at 0.86 seconds (compared to 0.70 seconds on HDD). However, there is a gain in elasped time. With SSD we see 97.16 seconds elapsed compared to 198.99 seconds for HDD. That means that for SSD we have twice the performance gain. We will in turn look at the wait times with SSD:
Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- -----------Disk file operations I/O 3 0.00 0.00 db file sequential read 23 0.00 0.03 direct path read 5276 0.06 29.11 direct path write 26010 0.04 58.69 ********************************************************************************

As table testwrites was based on tdash, we will read the index key values of tdash table into memory. Using an associative array, we will put object_ids from tdash table into this associative array. Using a cursor and a LIMIT clause to split the collection into chunks of 1000, we will process each chunk in turn. The code is shown below. Within the PL/SQL block, we perform UPDATE, DELETE AND INSERT operations on table testwrites, matching object_ids with 100 hundred randomized object_ids from tdash table. The FORALL keyword instructs the PL/SQL engine to bulk-bind input collections before sending them to the SQL engine. Together with BULK COLECT, FORALL is useful for bulk processing. In our case we are processing 100 rows at a time. Note that only statememt immediately after FORALL is executed. In other words, for the three DML operations below we will need three FORALL keywords. Each DML statement is followed by a commit.
ALTER SESSION SET TRACEFILE_IDENTIFIER = tester_create_test_bed_dml; EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>ALL_EXECUTIONS ); DECLARE type ObjIdArray is table of tdash.object_id%TYPE index by binary_integer; l_ids objIdArray; CURSOR c IS SELECT object_id FROM tdash; BEGIN OPEN c; LOOP BEGIN FETCH c BULK COLLECT INTO l_ids LIMIT 1000; FORALL rs in 1 .. l_ids.COUNT UPDATE testwrites SET PADDING1 = RPAD(y,4000,y) WHERE object_id = l_ids(rs); commit; FORALL rs in 1 .. l_ids.COUNT DELETE FROM testwrites WHERE object_id = l_ids(rs); commit; FORALL rs in 1 .. l_ids.COUNT INSERT INTO testwrites SELECT * FROM tdash t WHERE t.object_id = l_ids(rs); commit; EXIT WHEN C%NOTFOUND; EXCEPTION WHEN NO_DATA_FOUND THEN NULL; WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE(Transaction failed); END;

Note similar number of direct-path reads and direct-path writes with SSD as with HDD. The difference is due to Total Waited for SSD in comparison with HDD. In total for magnetic hard disks it had to wait 195.35 seconds, whereas this figure is less than half for solid state disks at 87.8 seconds, which is a gain of of 2.2 times. This comes about because of much faster initila seek time for SSD. To make this clearer, we will look at a typical direct-path write in the TRACE file for SSD:
WAIT #10: nam=direct path write ela= 2353 file number=6 first dba=1852384 block cnt=32 obj#=84454 tim=1337174590931628

continued on page 32
3rd Qtr 2012 Page 31

Use Case Comparison of Oracle 11g on Solid State Disks (Part Two) continued from page 31 END LOOP; CLOSE c; END;

If we just compare the 1,313 elapsed seconds from SSD with 4,321 elapsed seconds from HDD, we see that with SSD we have more than three times performance gain compared to that from HDD for a typical OLTP load.

First we will look at the data on HDD. We will forgo looking at the individual components of DML from the TKPROF reporting, and we will just consider the summary report for all recursive statements. Just to clarify: Nonrecursive statements are statements issued by the user. Recursive statements are statements executed by that nonrecursive call. Nonrecursive and recursive statements can be SQL or PL/SQL.
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse 52982 0.23 0.47 3 19 0 0 Execute 105787 141.17 4308.08 3564902 12708368 43266845 3729575 Fetch 21254 0.22 3.72 4380 29023 0 1733335 ------- ------ -------- ---------- ---------- ---------- ---------- ---------total 180023 141.63 4312.28 3569285 12737410 43266845 5462910 Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- -----------db file sequential read 3260950 1.57 3660.65 db file scattered read 55729 0.57 244.64 4321 elapsed seconds in trace file.

The Volume of Redos and Writes Generated by the OLTP Load Measuring the volume of read and write in Oracle can be achieved by using the Automatic Workload Repository (AWR). AWR is a comprehensive and powerful tool for monitoring Oracle performance. Unlike tracing, AWR is not enabled for the session; it is for the complete database. It also requires diagnostic licence. AWR, among other things, generates snapshots of key matrices, such as system and session statistics, cpu, waits, redo, writes and other time-model statistics. Most monitor tools work with intervals and snapshots, and AWR is no exception. Oracle provides the dbms_workload_repository package to define AWR snapshots that we will be using to measure the volume of redos and writes in our PL/SQL block. Our test code thus would be modified to:

Perform checkpoints immediately before and after PL/SQL block Manually take AWR snapshots before and after running PL/SQL block
The modified code is shown below:
ALTER SYSTEM CHECKPOINT; EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT(); DECLARE type ObjIdArray is table of tdash.object_id%TYPE index by binary_integer; l_ids objIdArray; CURSOR c IS SELECT object_id FROM tdash; BEGIN OPEN c; LOOP BEGIN FETCH c BULK COLLECT INTO l_ids LIMIT 100; FORALL rs in 1 .. l_ids.COUNT UPDATE testwrites SET PADDING1 = RPAD(y,4000,y) WHERE object_id = l_ids(rs); commit; FORALL rs in 1 .. l_ids.COUNT DELETE FROM testwrites WHERE object_id = l_ids(rs); commit; FORALL rs in 1 .. l_ids.COUNT INSERT INTO testwrites SELECT * FROM tdash t WHERE t.object_id = l_ids(rs); commit; EXIT WHEN C%NOTFOUND; EXCEPTION WHEN NO_DATA_FOUND THEN NULL; WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE(Transaction failed); END; END LOOP; CLOSE c; END; / ALTER SYSTEM CHECKPOINT; EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();

That 4,321 elapsed seconds includes 4,312.28 seconds from recursive statements plus 9.15 secobnds for nonrecursive statements (not shown). These recursive statements occurred because of dynamic storage extension. Dynamic storage extension occurred in our case because table testwrites and its index had to extend beyond their allocated space (that is, extents were allocated). The wait times included 3,660.65 seconds for db file sequential reads that are single block I/Os. There was 244.64 second waits for db file scattered reads that are multiblock I/Os. Multiblock I/Os are typically more than one block. The next test would be running the above PL/SQL code on the schema built on SSD. The results obtained were as follows:
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse 55058 0.34 0.52 3 19 0 0 Execute 108039 184.69 1302.58 3068410 13687217 44442290 3729737 Fetch 27429 0.34 1.51 4377 39353 0 1738082 ------- ------ -------- ---------- ---------- ---------- ---------- ---------total 190526 185.39 1304.62 3072790 13726589 44442290 5467819 Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- -----------db file sequential read 2812428 0.26 809.38 db file scattered read 48110 0.18 27.70 1313 elapsed seconds in trace file.

Now we see the elapsed time for SSD at 1,313 seconds, including 1,304.62 seconds for recursive statements plus 8.92 seconds from nonrecursive ones (again not shown). Note the wait times of 809.38 seconds and 27.70 seconds from db file sequential read and db file scattered read, respectively.

Once the above is run for both HDD and SSD, we will look at the AWR report generated between those two snapshots. Further information can be obtained from Oracle manuals that provide full documentation for AWR usage and reporting. Our interest is looking at various matrics that we could not get with the tkprof report. First we look at the AWR report for HDD. Under section titled Statistic we notice:

Page 32 3rd Qtr 2012

Statistic Total per Second per Trans -------------------------------- ------------------ -------------- ------------redo size 34,426,640,596 7,190,384.5 901,150.2 user commits 38,203 8.0 1.0

As before, we watch the Av(erage) Writes/s. For SSD it gives 2,826 physical I/O writes per second, which is significantly higher than that for SSD at 292 physical I/Os per second. Although it is interesting to look at these figures, Oracle user processes do not need to wait for a checkpoint. The writes are done asynchronously, and they are background events. Thus they do not count in user response times. Going back to elapsed times from TKPROF report, for a mixed OLTP load we gain typicaly three times better performance for OLTP writes with SSD compared to HDD.

Note the Per Trans(action) values. There were 38,203 commits and one commit per transaction. So for every transaction there were 901,150.2 bytes of redos. If we multiply per Trans rates with the number of user commits we will get the Total bytes. There were 901,150.2 * 38,203 = 3.4427E+10 bytes or 32GB of redos. Since the HDD-based table and index reside in the APP1_HDD tablespace and the SSD-based table and index reside in the APP1 tablespace, we will consider I/O performance for HDD first. This is tablespace APP1_HDD. The writes and average (Av) writes/s are in I/O not blocks:
Tablespace -----------------------------Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) -------------- ------- ------- ------- ------------ -------- ---------- ------APP1_HDD 3,535,159 738 1.2 1.1 1,396,750 292 0 0.0

Conclusion In the second part of this article, we looked at the use case for solid state disks for write-intensive activities in Oracle 11g Release 2. We looked at direct-path writes for SSD and concluded that it performs reasonably better compared to HDD but not that much faster. The performance gain we got was around twice with SSD. We then looked at a typical OLTP load with index, and the conclusion was that with random access, SSD performs at least three times better compared to HDD.
About the Author

The segments I/O stats are given below. These physical writes are in blocks, and I have added the last column to give the size in MB.
Tablespace Obj. Physical Physical Name Object Name Type Writes Writes/MB ---------- -------------------- ----- ------------ ----------APP1_HDD TESTWRITES TABLE 4,310,514 33,676 APP1_HDD TESTWRITES_PK INDEX 3,730 29

Mich Talebzadeh is an award winning consultant and a technical architect who has worked with database management systems since his student days at Imperial College, University of London, where he obtained his Ph.D. in experimental particle physics. He specializes in the strategic use of Sybase and Oracle. Talebzadeh can be reached at C mich@peridale.co.uk.

The matric to watch is the Av(erage) Writes/s. This gives the thoughput of writing data to segments on HDD. It gives 292 physical I/O writes per second Now we turn our attention to schema on SSD. Under section Load Profile wehave:
Statistic Total per Second per Trans -------------------------------- ------------------ -------------- -----------redo size 34,413,294,816 27,284,637.6 916,465.9 user commits 37,550 29.8 1.0

Working out as before, there were 916,465.9 * 37,550 = 3.4413E+10bytes or 32GB of redos. So we had the same volume of redos as HDD. Take a look at the tablespace used for SSD tests. It is called APP1. The stats show:
Tablespace -----------------------------Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) -------------- ------- ------- ------- ------------ -------- ---------- ------APP1 2,860,666 2,268 0.3 1.1 3,564,644 2,826 0 0.0 And the segments I/O stats as before: Tablespace Obj. Physical Physical Name Object Name Type Writes Writes/MB ---------- -------------------- ----- ------------ ----------APP1 TESTWRITES TABLE 4,556,094 35,594 APP1 TESTWRITES_PK INDEX 3,693 28

Advertisers Index
The International Oracle Users Group and SELECT Journal would like to thank the following members of the community for their support. For more information about their products and services, visit their Web sites. And be sure to let them know you saw them in SELECT Journal!

Advertiser . . . . . . . . . . . . . . . . Web Address . . . . . . . . . . . . . . . Page Number CISCO . . . . . . . . . . . . . . . . . . . . . . . . www.cisco.com. . . . . . . . . . . . . . . . . . .Inside Front Cover Embarcadero . . . . . . . . . . . . . . . . . . . www.embarcadero.com . . . . . . . . . . . . .4 EMC . . . . . . . . . . . . . . . . . . . . . . . . . www.emc.com . . . . . . . . . . . . . . . . . . .12 GridIron. . . . . . . . . . . . . . . . . . . . . . . www.gridironsystems.com . . . . . . . . . . .3 SAP . . . . . . . . . . . . . . . . . . . . . . . . . www.sap.com . . . . . . . . . . . . . . . . . . .Back Cover STEC. . . . . . . . . . . . . . . . . . . . . . . . . http://stec-inc.com . . . . . . . . . . . . . . .Inside Back Cover

3rd Qtr 2012 Page 33

SELECT Star

New Lock Management, COLLABORATE 2012 and the Oracle Cloud

ratios were obtained when the data were presorted.

The operation needs to be direct-path loaded (APPEND, PARALLEL, Direct

Load, etc.) for HCC.

Updating any rows that are compressed using HCC will mean they will be
row migrated and not HCC compressed. In fact, it is OLTP compressed. rows within it will be locked, too.

When an HCC row is locked, the HCC compression unit (CU) and all the For a table with HCC compressed rows, dropping a column does not
remove the column immediately; it tags the column as not available and removes it at a later time. later time.

To speed up data loads, load the data uncompressed and compress it at a I/O Resource Manager (IORM) can be applied within a database (intra) as
well as between databases (inter).

By Tony Jambu

nother great COLLABORATE conference was held this year in Las Vegas where not a single day during the five-day conference was below 85 degrees Fahrenheit.

Read the quarterly bundle patches carefully. Then read them again. Go
As with most technologies, this one has its own share of challenges. Listed below are some that were experienced by existing Exadata sites:

over it three times. That was the advice from the Exadata discussion panel. It is best to wait a few weeks before applying the patches.

In addition to the hot topics of virtualization and cloud computing, Oracleengineered systems, data warehouse and business intelligence, MySQL and performance tuning, we saw an increased number of sessions on big data, Oracle WebCenter and Oracle application specific sessions. If you thought virtualization and cloud computing were a fad last year, now you know that is not the case. Virtualization and cloud computing are here to stay, and this year we got to hear from companies that have embarked on and embraced the technology. DBAs will need to learn how to deal with and make the most of these key technologies or they will be left behind.

Managing the dynamic parallelism of Oracle 11g Implementing huge pages Difficulty with Dynamic Resource Management (DRM) and possibly
disabling it

Understanding quarterly bundle patches

New Hang Management Framework Introduced in 11.2.0.2 If you come across the following error in you alert log, you are probably wondering what it means:
ORA-32701 Possible hangs up to hang ID=0 detected

The Rise of the Exa Machines The No. 1 hot topic was still the Oracle Exadata machine, but this year it also encompassed other engineered system such as the Exalogic and Exalytics machines. The term being coined regularly around the conference was The Rise of the Exa Machines.
The following are tips and hints gathered from various sessions and forums. Exadata X2 has upgraded its high capacity SAS disk from 2TB to 3TB still running at 7200 RPM with no price differences. The ROI for an Exadata machine is more than three years with 50-65 percent saving in storage requirements for some clients. For migration to Exadata, it is best to perform it in two phases:

It appears that, beginning from Oracle 11.2.0.2, Oracle introduced a new hang management framework. This new framework would monitor the database. If it finds a session that is blocking multiple sessions, it would then kill the blocking session.
_hang_hiprior_session_attribute_list _hang_ignored_hang_count _hang_ignored_hangs_interval _hang_log_incidents _hang_long_wait_time_threshold _hang_lws_file_count _hang_lws_file_time_limit _hang_msg_checksum_enabled _hang_output_suspected_hangs 1 300 FALSE 0 5 3600 TRUE FALSE Hang Management high priority session attribute list Hang Management ignored hang count Time in seconds ignored hangs must persist after verification Hang Manager incident logging Long session wait time threshold in seconds Number of trace files for long waiting sessions Timespan in seconds for current long waiting session trace file enable hang graph message checksum Hang Management enabled output of suspected hangs

Phase 1: Migrate the applications as is. There is no need to make any

changes, as the applications, on average, will be 30-35 percent faster.

Phase 2: Investigate the removal of unnecessary indexes by first

making them invisible using the ALTER INDEX index_name INVISIBLE command. You can verify that the index is not invisible by querying [DBA|USERS|ALL]_INDEXES.VISIBILITY column.

Next, consider removing performance hints. Other tips:

When implementing hybrid columnar compression, the best compression

Page 34 3rd Qtr 2012

_hang_resolution_confidence_ promotion _hang_resolution_global_hang_ confidence_promotion _hang_resolution_policy _hang_resolution_promote_ process_termination _hang_resolution_scope _hang_signature_list_match_ output_frequency _hang_statistics_collection_interval _hang_statistics_collection_ma_ alpha _hang_statistics_high_io_ percentage_threshold _hang_verification_interval

FALSE FALSE HIGH FALSE

Hang Management hang resolution confidence promotion Hang Management hang resolution global hang confidence promotion Hang Management hang resolution policy Hang Management hang resolution promote process termination Hang Signature List matched output frequency Hang Management statistics collection interval in seconds Hang Management statistics collection moving average alpha Hang Management statistics high IO percentage threshold Hang Management verification interval in seconds

It is a free tool and can be downloaded from the link mentioned earlier. Ifyou need some help, a cookbook exists, written by Karl Arao: http://karlarao.tiddlyspot.com/.

PROCESS Hang Management hang resolution scope 10 15 30 45 46

Quick Tip: Finding the Minimum, Average and Maximum Throughput of Oracle Backup Have you ever wanted to know the minimum, average or maximum throughput of your Oracle backups? Daryl E. from the Oracle forums website provided such a solution: https://forums.oracle.com/forums/thread.jspa?t hreadID=2147313&tstart=180.
SELECT MIN(ROUND(Effective_bytes_per_second*60*60/1024/1024/1024,0))Min_gbperh, ROUND(AVG(ROUND(Effective_bytes_per_second*60*60/1024/1024/1024,0)) )Avg_ gbperh, MAX(ROUND(Effective_bytes_per_second*60*60/1024/1024/1024,0))Max_gbperh FROM V$backup_async_io;

MIN_GBPERH AVG_GBPERH MAX_GBPERH 0 54 1873

It is possible to turn off this feature or even change its behavior by changing some of the hidden parameters shown above. However, I have yet to test this out. This should not be tested against a production database without Oracle support being involved.

Depending on your system, you may need to use the synchronize view instead: V$BACKUP_SYNC_IO instead of V$BACKUP_ASYNC_IO.

SLOB: Silly Little Oracle Benchmark Not another benchmark utility for Oracle! This is one with a difference, though. Sure, there are free benchmarking tools such as Hammerora and SwingBench or the commercial option Benchmark Factory from Quest Software. These are proper benchmark tools. They generate OLTP/DW workloads, and the results are measured by transactions per second. But what if you are interested in IOPs? What are your options?
The Silly Little Oracle Benchmark or SLOB by Kevin Closson, technology director and performance architect in the Data Computing Division of EMC, is an option. Closson was Oracles performance architect in Oracle Corporations server technology group Exadata development organization and has since moved on to EMC. Closson developed SLOB over a number of years going back to the 1990s. On his website, http://kevinclosson.wordpress.com/2012/02/06/introducingslob-the-silly-little-oracle-benchmark/, he points out that SLOB is neither a benchmark tool nor is it silly. So what is it? From his website: SLOB possesses the following characteristics: 1. SLOB supports testing Oracle logical read (SGA buffer gets) scaling 2. SLOB supports testing physical random single-block reads (db file sequential read) 3. SLOB supports testing random single block writes (DBWR flushing capacity) 4. SLOB supports testing extreme REDO logging I/O 5. SLOB consists of simple PL/SQL 6. SLOB is entirely free of all application contention

Select Star Mailing List For those who are already on the Select Star mailing list, most of the above information may be old news to you. For those of you who would like to get the latest news as soon as it occurs, why not subscribe to the mailing list? It contains information that is not published in this magazine.
If you would like to receive similar information, subscribe to the mailing list by either visiting the URL http://groups.yahoo.com/group/Select_Star/ or emailing Select_Star-subscribe@yahoogroups.com. Onaverage, there are one or two postings per week. If you have any comments or contributions, please send an email to Select_Star@yahoogroups.com.

About the Author

Tony Jambu has more than 19 years of Oracle experience as a development DBA and database architect. Jambu has an in-depth working knowledge of Oracle and its breadth of technology and unreservedly shares his knowledge with others by presenting regularly both internationally and in Australia, winning best presentations at various conferences. He has been the author of the SELECT Star column for IOUGs SELECT Journal since the magazines inception in 1993. Jambu was awarded the Oracle Consultant of the Year by Oracle Magazine and Oracle ACE by Oracle Corporation for his knowledge and experience in Oracle technology as well as for his technical contribution to the Oracle community.

3rd Qtr 2012 Page 35

Oracle ACE

The Oracle ACE program recognizes excellence in the Oracle technology and applications communities, and rewards individuals who generously share their technical knowledge and experiences. Learn more about the Oracle ACE Program.

Interview with an Oracle ACE: Michelle Malcher

switched around careers about 10 years before landing in technology, but I guess the different perspective from that experience has also made me well-rounded in my career.

IOUG: Do you have any advice for novices in this industry? Malcher: Get ready to keep learning! The fact that companies have a lot of data seems to be one constant, and there are so many opportunities to do things with that data. There is a lot to be said for experience, so find ways to test and play with features in a sandbox environment. Be ready to fix (admit to) mistakes and be open to new ideas. IOUG: Do you have any advice for IOUG members for their own careers? Malcher: Stay involved in the community. I have always been able to learn more and understand more about database technologies by sharing information and presenting. Having a network of people that have similar experiences and a place to go to bounce ideas off of is extremely important in building a stronger knowledge base and stronger career path.

IOUG: When did you become an Oracle ACE? Malcher: I became an Oracle ACE in 2006 and became an Oracle ACE director in 2009. IOUG: What does this experience mean for you personally and professionally? Malcher: I felt a great sense of accomplishment and was honored to be nominated to ACE director. This is a great recognition from Oracle that I have done some pretty cool things with Oracle and that I am willing to bring others along with me to understand databases and the challenges we face as database professionals. This also provided new opportunities to learn what Oracle is working on and provide feedback of how the technologies are being used. IOUG: Has your status as an Oracle ACE helped you in your career? Malcher: I believe it has opened up new opportunities to present, and being considered an expert in an area at work does have advantages. But if the knowledge and actions dont back up the title, then it wouldnt matter if I was an ACE or not. IOUG: In your current role researching, writing and teaching for Oracle professionals, what has been your biggest achievement? What has been your biggest regret? Malcher: I think being able to come to work and enjoy what I do is a big achievement. I really enjoy working with databases and all of the aspects around the environment. The additional opportunities to write and teach surprise even me. I regret not starting my database career sooner. I had

Michelle Malcher presents

Oracle Database Security Tip #243

I really dont know what tip number this is, but the tip number doesnt really matter. In the area of security with all of the different options out there, it might be easy to forget about the simple things, such as granting extra permissions for a one-off task and forgetting to remove them. Monitoring the permissions granted and auditing changes are the key here for making sure the permissions are changed back. Auditing is turned on by default in Oracle 11g and GRANTS against new tables are audited, so the audit trails will help in managing the users access to objects. Auditing system privileges and those privileges with ANY object type of permissions would be another area. Just doing a simple select against the dictionary tables, SELECT GRANTEE from DBA_ROLE_PRIVS where GRANTED_ROLE=DBA would be a good check to make sure applications logins havent slipped in because of anupgrade.

Page 36 3rd Qtr 2012

Accelerate Access to Your Oracle Databases

Our solid-state solutions speed the performance of your most demanding Oracle database applications.
Welcome to the world of big data, analytics and virtualization. With 2.7 zettabytes of data being generated and accessed in 2012,* and an ongoing rush to get it crunched quickly, lightning-fast access to data is paramount. But theres no longer a need to throw more money at HDDs to accommodate this growth. Setting a new standard for enterprise database application performance with high transaction rates and vastly superior latency, STECs PCIe, SAS and SATA enterprise SSDs are the industry game changers designed to supercharge your data center. STEC SSDs accelerate enterprise software applicationsdelivering the performance, reliability and improved cost per IO required in todays robust enterprise. From our PCIe SSDs for servers and ZeusIOPS SAS SSDs for SAN, NAS and DAS systems, to the MACH16 SATA drives that deliver cost-effective SSD capacity, our solid-state solutions are easily integrated for high transaction volumes and super-low latencies. And because were qualified at most major OEM storage and server suppliers, you can trust STEC to help accelerate your mission-critical applications. Visit www.stec-inc.com/ioug today to learn about how our solid-state solutions deliver better ROI and can help lower TCO with your Oracle database application deployments.

2012 STEC, Inc. All rights reserved.

*Source: IDC

Accelerate Access to Data

190,5x254_SAP=EU11-10-10_Hardrock_39l.indd 1

2011 SAP AG; SAP and the SAP logo are trademarks and registered trademarks of SAP AG in Germany and several other countries. O&M SAP EU 11/10

28.02.2011 12:28:24 Uhr