The Art of Being a Successful DBA – Good

Documentation is Essential
Let’s start our series on Art of Being a Successful DBA by covering the art of good
documentation. Although the importance of a well thought out and detailed documentation
library is blatanly obvious, creating documentation is the task most often postponed by an
overworked DBA unit.

Documenting processes, procedures and best practices is a task that is often considered to be
boring and mundane. Most DBAs would rather perform virtually any other activity than sit in
front of a screen using a word processor. As a result, creating documentation is often postponed
until the DBA has a little free time to kill. Today’s database administration units are operating
with smaller staffs, tighter budgets and ever-increasing workloads. The end result is that the
documentation is either never created or created and not kept current.

But a robust detailed documentation library creates an environment that is less complex, less
error-prone, reduces the amount of time DBAs spend learning new database environments and
reduces the overall time spent on day-to-day support activities. DBAs are able to spend more
time administering the environment than finding the objects they are trying to support and the
processes and programs used to administer them.

The nature of my business as a remote services provider demands excellent documentation. The
majority of environments we administer weren’t designed by my organization. The only way that
we can ensure high quality and high-speed administration of these environments is to document
them thoroughly. We document everything, from initial connectivity and customer contact sheets
to detailed information on batch job streams and individual program execution. If we need to be
aware of it, we have it documented. We have dozens (and dozens) of internal best practices also
documented. If you are evaluating remote database services providers, you need to ask them for
examples.

Documentation is also the foundation of many of the other disciplines I will be discussing in
future blogs. Let’s continue our discussion with a few helpful hints to get you started.

Understanding the Important Role Good Documentation Plays

We all generally understand the benefits that documentation provides. I think that all readers will
see the importance I personally place on documentation in upcoming blogs. Virtually every topic
that I will discuss, from repeatable processes to paranoid administration best practices, I will be
mentioning what to document and the important role that documentation plays.

Let me reaffirm my opinion in this one sentence:

Documentation is one of the foundations for virtually everything I will be discussing in this
series on The Non-Technical Art of Being a Successful DBA. Enough said.
Creating an Organizational Environment That Fosters Good Documentation
I have been leading database teams off and on for 15 years now. It is my responsibility as
manager to create an environment that fosters the production of robust and high quality
documentation. Let me describe some of the challenges that I have faced in the past and how I
have overcome them.

Since I view high quality documentation to be my responsibility as the director of database

services here at Remote DBA Experts, I ensure that it becomes part and parcel of every DBA’s
job description. If we feel that a candidate DBA doesn’t value the importance of documentation,
we don’t hire them. It’s OK if they don’t like to do it, they just need to express that it is
something that is required to create a high-quality support environment.

I will add time for documentation when I estimate the amount of time it will take me to perform
an administrative task during project planning meetings. I don’t settle for “we can do that after
the project is complete” as an answer from an application project manager.

If I get overruled and lose one battle, I still make sure I document my activities after the project’s
critical path is complete. And I don’t let a single setback prevent me from telling everyone in the
next planning meeting that I need so many hours to document what I have done.

If you continuously sell the importance of documentation sooner or later you will begin to wear
your opponent’s down. Although I prefer to call it “being relentless”, I’m sure that many of the
application development managers (and my own managers) viewed it as “being a ….” (insert
your favorite description here).

Every document I have created that provides a list of activities I , or my unit, needs to perform
during a project has documentation included. It helps to integrate it into the fabric and culture of
my organization’s environment.

Making Documentation Second Nature

You also need to ensure that generating documentation becomes a natural part of your daily
activities. You must continuously remind yourself that documentation is a primary and integral
part of providing high quality support services to your customers.

You must also remind yourself that it makes your job easier and benefits your fellow DBAs. You
know what has really irritated me the most during my career? When a fellow DBA needs to be
out of the office for a time and asks me to “help them out” by performing a complex, application
specific administrative activity and then tries to verbally tell me how to perform the 300 steps it
takes to do it.

Ever try to refresh an ERP application test environment from production when that test
environment doesn’t have enough space to hold all of production’s data? 4,000 steps later and
you begin to second-guess your choice of professions. That was the exact request from one of
my fellow DBAs a few years ago when I worked for a large aluminum producer here in
Pittsburgh. Not only did he get me to do the refresh but I also had to document the process for
him along the way. Some call that being a good coworker; I would view that as having a big
sucker taped to my forehead. Live and learn, I guess.

The moral of this story is: If you don’t want to be the only one that can perform that 900 step
ERP Application production to test refresh, document it! If you don’t want to be called by the
on-call DBA because he doesn’t know exactly where to add a datafile in an emergency situation
(like an application developer forgetting to tell youu that they were loading 10 million additional
rows into that 100 row table), document it! The more you document, the easier your life as a
DBA becomes.

I’ve never had a great memory. It makes documentation easy for me. I also like to write and that
helps. But I will admit that there are times that I would rather perform virtually any other activity
than document.

But it has become easier because I continuously reaffirm to myself the importance of
documentation. The more you reinforce that to yourself, the more second nature (and easier) it
becomes.

Making Documentation Easy

I’m a huge fan of documentation templates. Here at Remote DBA Experts we have, or are in the
process of creating, templates for everything we document. We have templates for documenting
connections to our customer’s environments, their backup and recovery environments, their
application specific processes to name a few. If it needs to be documented on a regular basis, we
have a template for it. We also have generic templates for documenting environments and
activities that don’t fit into other templates.

Microsoft Word document templates provide many features that streamline the documentation
process and help to improve the quality of the content they store. I try to take advantage of as
many features as I can. I use drop down selection menus, check boxes and radio push buttons to
improve the speed and quality of the documentation process. I also take advantage of the help
pop-up feature that Microsoft Word provides to create a detailed description of what information
is to be entered into that field, check box or radio button.

I created my first template close to 15 years ago. It wasn’t created to document a process, but to
document a change request. I’ll be covering change management in an upcoming series in this
blog. Was the document crude? You bet. But it did the job. It has gone through numerous
iterations and migrations from one Word release to the next. I continue to use it at organizations
that don’t have a formalized and documented change management process. And you can’t
believe how many I have come across that don’t.

There are dozens of software companies that offer content management solutions. Database
vendors have also recognized this as a lucrative market. All of the major database vendors now
offer advanced content management software, each one trying to outdo the other in the number
of bells and whistles that their product’s offer.
Do a quick search on Google for documentation content management software and you will find
out just how many competing products there are.

Content management products offer check-in/check-out features, document versioning, web

portal access and advanced workflow capabilities to name just a few of the features desgined to
improve content management. The competition in the content management market space is fierce
to say the least. Content management vendors know that continuously adding new bells and
whistles to their products is not just important for increasing market share; it is critical for their
survival. Product costs can range from thousands to tens of thousands of dollars (or more).

If you have the funds and your management understands the benefits that a full-blown content
management package provides, by all means begin a content management product analysis. But
if you don’t have the funds, create a shared drive on your network and declare it to be the “DBA
Documentation Portal.” We will be using a combination of Microsoft Sharepoint and Sales Force
as our own DBA Documentation Portal. Sales Force’s interaction with our document libraries
provides us with a highly secured, highly customizable document storage environment. It is our
“one stop shop” for customer information.

What to Document
By all means this is not an all-inclusive list of what can be documented. Consider it as a starter
kit to help you begin your quest for “documentis nirvanas”. Is some of this overkill for your
particular environment? Maybe, but just consider this a general, high-level list.

Database Environment Documentation

 Naming conventions
 Servers (server names, operating system release, hardware vendor)
 Databases (vendor, database version, features enabled)
 Directories (everything from Oracle software installations that don’t follow Oracle
standards, application code storage to personal script locations). If you, or a fellow DBA
needs to find it, document where it is!

Application Specific Documentation

 Application type (i.e. data warehouse, online transaction processing, decision support,
third-party application name and functionality it provides).
 Business unit requirements and related information for supported databases
 Uptime requirements (i.e. 24 X 7, 8 X 5)
 Database downtime windows
 Critical job processes
 Business unit and application developer contact lists
 Turnover windows for database changes
 Problem notification and escalation procedures
 Security Sensitivity. How sensitive is the data?

Process Documentation
  Repeatable administrative processes (covered in an upcoming blog)
 Backups – Probably the most critical set of documentation you will ever create.
Document how it is backed up, what scripts back it up, where the back up is going to,
tape retentions and backup message directories. If it is involved with a backup,
DOCUMENT IT. Review the document with other units that are involved in the backup
and recovery process. It is your responsibility to ensure that you don’t hear an operator
say “What retention period? Nobody told me we were to have a retention on these tapes.”
when you are in a recovery situation. Remember that Oracle states that human error,
including miscommunications, is responsible for over 90% of failed recoveries. If you
want to reduce recovery failures DOCUMENT THE PROCESS AND REVIEW IT.
 Oracle utilities. Document Oracle Import/Export input and output directories and where
the files are that execute the utilities. The same with Oracle’s load utility. Document the
execution scripts and input and output directories. A short note on why you are running it
is also required
 Anything else you run on a regular basis to support a specific application
 Change Management. I’ll be spending an entire blog, or two, on this
 A daily monitoring activity checklist to ensure that no activity is missed. We have daily,
weekly and monthly activities that are to be performed for each of our customers
 Assuming on-call responsibilities. You need to make sure you are prepared for the on-call
process. The checklist should include what other DBAs are working on during your on
call rotation, customer outages taking place, connecting to environments that are hard to
connect to, etc..
 Complex administrative activities performed regularly
 Test and reporting database refreshes
 Data reorganizations
 Disaster recovery tests. The processes required to perform the recovery AND the criteria
that will be used to evaluate whether it was successful or not

Object Documentation

 DBA specific stored PL/SQL programs

 Day-to-day support scripts (where they are and what they do)
 Monitoring scripts (where they are and what they do)
 Scripts used to perform database administrative changes. I have specific directories that
provide output from critical database changes that I have performed and other directories
containing the SQL used to make that change
 Operating system scripts. Document what the script does in the beginning of each of your
scripts. Ever try to determine what a 400 line script does that was created by someone
that knows much more about UNIX scripting than you do? We have all been in that
position at one time or another during our career. I’m personally still not good at it.
Which is why I never was a great developer (all that IF-THEN-ELSE-OTHERWISE-
GOTO-EXIT stuff bored me to tears). Make it easy on your coworkers to understand
what the script does by putting comments at the top of the script as well as in the body.
Also keep a running history of script changes. What they were and the time they were
made
Database Administration Unit Organizational Documentation

 Contact Information
 DBA responsibilities. Which databases and tasks they are responsible for
 DBA unavailablity. Allows application developers to plan for a DBA not being available

It is a good practice to distribute this information to all business units supported by the database
administration unit

I hope you enjoyed this blog on documentation and the important role it plays in the Art of Being
a Successful DBA.