Knowledge Base 6418:

Planning your first 90 days as the new Remedy administrator

Adopting a new role and supporting new applications can be a challenging experience, especially when
there are unknowns about how and why things work as they do. The new administrator is justifiably
conservative, afraid to make changes for fear of what it may break, and mostly reacting to problems as
they appear. In contrast, the experienced Remedy administrator, if properly prepared, can be a sage
agent of change within your organization, recommending, evaluating, and championing improvements in
your system to reflect your evolving business needs. There is no single right path for going from one to
the other, but a frequent element in the successful journey is a bit of up front planning, evaluating what
roles will be required of the Remedy administrator and how best to facilitate them. A further reward of
this exercise is that its outputs also facilitate training for your backup administrator, or transitioning the
responsibilities to your replacement when you move on.

The paragraphs below discuss some ideas that you may want to consider in your development plan, and
references additional resources, which may be helpful.

The good news is that being an AR System administrator is similar in practice to being administrator of
other applications. The same traits which make a successful administrator are applicable: being thorough
when investigating changes, being responsive to problems reported, deploying changes in an orderly
fashion, following a plan of preventative maintenance, and preparing for disaster recovery.

Define your current environment

A good first step in assessing your needs is to define your present environment. This step may not be
necessary if you have been involved from the beginning, but in most cases it's good to come up to speed
on developments previous to your involvement. This may include putting together a summary sheet of
application and product versions, collecting any requirements definitions, design docs or logs maintained
either by your predecessor and/or consultants. Also gather product manuals, media, internal
documentation, and training materials.

These are very tangible components and specific details. Another aspect of defining your current
environment involves understanding the role of these applications within your organization, what needs
they fulfill, what constituencies they serve, and how these will grow and change over time. What
features or roles of the application are areas of upcoming development? When evaluating your current
environment, try to get a grasp of both the size and shape of the application and administrative role, as
well as a feel for its progression within your organization.

Define your roles and needs to fulfill them

The next step of the evaluation should investigate your needs to successfully perform your administrative
duties. Below are some common administrator responsibilities.

- Application configuration such as adding categories, groups, etc

- Preventative maintenance, such as performance tuning, archiving
- Addressing reported problems by users
- Investigating and implementing application customization (in some cases)
- Deploying application changes or upgrades
- Backup/restore process

This needs assessment step is an important part of preparation, and should be based upon your role and
current environment as discovered earlier rather than a generic list, but below are some suggestions to
start with:
Familiarity with configuration tools
Need to know what configuration steps are interdependent, or related, what the tradeoff in choosing one
or the other
How do the different parts of the application work together?
Are there different modes or roles in the application?
What diagnostics are available to troubleshoot problems?
What diagnostic modes are available?
Where is the application and configuration data stored?
What tools do I need?

In preparing a list of needs, try to organize them into three categories: things you need to know from
memory, things that can be available as reference material, and tasks or things you'll have to do once to
be prepared.

Collect and assess resources

Once you've defined your needs, it's time to evaluate and collect resources to address them. These two
steps, defining your needs and evaluating available resources, are often done jointly as part of general
“looking around”, but arguably you'll get a more complete list of your needs by making that list first,
without letting the most available resources steer your impression of your needs.

Generally available resources include documentation manuals, education courses, knowledge base
entries, and a variety of other technical resources available from the Supportweb web site.

Electronic versions of the manuals are generally available for download from SupportWeb, where you can
right click on the link and save the PDF file locally. Quite possibly, there will be additional manuals that
are related, either high-level conceptual overviews or advanced topics such as the Programmer's Guide,
which you will not have in hardcopy so perusing the Documentation area of the website is a good way to
look at what resources are available.

Likewise, course descriptions are available via the web site. In considering what courses to attend, go
back to your summary of needs, both in your roles and your categorization of needs (what you need to
know from memory, vs. available as reference, vs. having performed at least once.) Courses provide
coverage of the most frequently needed information, and provide a sound big picture of topics, laying
down a foundation that will be augmented by the manuals and experience administering the system.

Let's say that in your evaluation of your current environment, needs and resources, you determined that
your position will be an application administrator for the HelpDesk application, which will initially be
deployed and administered pretty much out-of-box. But within the next few months after deployment,
your role will grow to collect requested changes, to prioritize and implement them, and then to begin
work on implementing the full Asset Management application. In the review of the available manuals and
resources for the HelpDesk and Asset Management applications, and the Action Request System for those
customizations, you found the former to be too point-specific to determine the best way to approach
configuration, and the latter way too detailed to know where to start.

Since your initial role would be that of an application administrator, you may decide to attend the
HelpDesk course first, or that one plus the Asset Management course, to get a feel for both though your
focus on Asset will not come until later. The evaluation of your needs, assessment of other resources,
and some preliminary thoughts on how to set it up, would make for a productive class as you would be
able to address this questions during that time. Suggestions and ideas from the instructor or your
classmates can provide insights if you come to class after a little thought into the issues.
Having decided to not attend the “Administering the Action Request System” or “Advanced Topics”
courses just yet, you would exercise a bit of restraint in using those tools to make any customizations.
That is, you would focus on other administrative duties and diagnostics, collecting requests and
evaluating whether they can be done within the HelpDesk application itself. Perhaps a few months later,
you would schedule these additional courses on development in the Action Request System, and armed
with some specific requests and your own experience, you'd get a lot out of the courses. This is just one
approach, and others of course exist, but it's an example of how to let your needs and your timeline
determine what courses to take, when to take them, and how to get the most out of the experience.

Finally, as part of this step of familiarizing yourself with available resources, acquaint yourself with how to
access Technical Support and the specifics of your Support Plan, as well as internal system support
personnel like your system administrator or DBA. During what hours can you reach these resources, and
can you reach them after hours? How should your report problems or make requests of these resources?
What information will they need when you report problems to them?

Make a “To do” list

The last step in this process is to write a preliminary 90 days timeline for your preparation as the new
Remedy administrator. Your timeline might include weekly tasks, or a “focus for the week”, and should
include target dates for each task. Be sure to schedule classes early enough that you can position them
appropriately in your plan. The discussion above and the steps you've taken for each step should you
prepare to make an informed plan. Like all plans, it is subject to unknowns that can change it. Also, the
90 days timeframe if fairly arbitrary, it could be as short as 30 days, but be sure to give yourself sufficient
time. Consider how much time you'll have available for your administrative duties, if you also perform
other roles for your organization.

Remember that the objective is that at the completion of your plan, you'll be confident and accomplished
in your role of administrator, can efficiently and eagerly address problem reports and change requests,
able to broadly address questions and suggestions about the abilities of the system and your
implementation to address your organizational needs. So write your plan to accomplish that goal.

Also see the following knowledgebase article for some ideas in preparing your plan:

KB 6366: Suggestions for setting up a maintenance plan

A couple of additional resources of interest to new administrators:

On Diagnostics:
KB 6278: A primer on reading filter and workflow logs
KB 6161: Setting up a dev server, how and why

On Customization:
KB 6216: A checklist for unit testing application customizations
KB 6196: Suggestions for rolling out application customization
KB 5451: Customization requests addressed to Support vs. Consulting
Knowledge Base 6366:
Suggestions for setting up a maintenance plan

As many experienced administrators will attest, having a sound maintenance plan is key to both
preventing problems and resolving them quickly when they occur. Below are some suggestions for the
new Remedy administrator in putting together a maintenance plan.

A starting maintenance plan like the one suggested below might be 5 to 8 pages in length, and be used
both as a checklist of periodic duties and a place to collect important information on your system
environment. Such a document can be a big help to your backup administrator, and facilitate working
with tech support, your system administrator, DBA, network administrator, or other staff who are take
care of your systems.

1. Summary sheet of application and tool versions

Troubleshooting often begins with specifying the versions of each component, so this will save a lot of
time. The suggestion here is to keep a summary sheet of your current versions, along with how you find
the version number. For example, you can check the version of most client tools by choosing a menu
option, Help | About.

For other suggestions on checking versions see Knowledge Base (KB) entries: 5630, 5469, 5407, 5375,
5376, 5377, 4962, 4462, 2866

2. Summary sheet of startup locations, scripts, etc

For similar reasons, you want to record the installation directory of products on the server, and any
startup processes or scripts that are part of the application. This helps to map what components on the
system should be protected, backed up, and/or shared, and is especially useful if the system is shared
with other applications.

See also KB 4177 for an example on the HelpDesk application.

3. System specs and names

The server name, fully qualified domain name, IP address, version of OS, server model, memory, disk
space, and similar specifications on the AR Server and database server may be useful at different times
for troubleshooting.

4. Disaster recovery/backup plan

Arguably the most important responsibility of an administrator is to be able to restore the system to a
working state if some massive problem occurs. It's a rare occurrence, with Remedy or any other system,
but precautions should be taken to ensure it can be done smoothly. If working with a DBA or system
administrator, you'd want to ensure that they have all the information they need and that your roles are
clearly defined.

Test your backup plan by restoring your production system to a test server to make sure it works as
expected, and thoroughly compare behavior with your production server. Remember that backing up
your database is of no value, unless you can successfully restore it, so a little practice is in order.
Restoring your backup to a test server (instead of your production server) allows you to test without
compromising your production server, validates your solution in the “loss of server” scenario, and can be
very useful in testing customizations on your test server by mimicking production.
Consult your DBA or system administrator regarding database and system backup strategy and DBMS
tools available.

5. Check diagnostic queues or logs

Several applications include diagnostic queues and/or logs. For example, the HelpDesk application writes
files to the RappDir directory before processing them, and similarly, the CRM Solutions applications write
records to the Run Process Queue form for processing. Checking the directory or form for collecting
files/records, or failures when processing them, is a good idea to identify problems early. The ARServer
maintains the ARError.log file for server errors, and your web server or database server may also have
log files that are worth checking periodically.

6. Check available space on servers

Your database may allow for automatic growth to accommodate data, but you would still want to
periodically check system drives for available space. A common problem to watch for is running out of
swap space or space on the drive containing the Temp directory, which can cause a variety of strange
symptoms. Your system administrator may have a more complete list of system checks.

7. Server restart procedure

To ensure you have all the steps, sequence, and checks all in one place, it's a good idea to write a server
restart procedure for yourself so it is available when necessary. Typically, the process involves shutting
down the services/daemons in a specified order, such as application services (rappsvc, bbqueued, etc)
then AR server processes (arserverd, etc), and then database services if all on the same system. There
may also be a step of verifying that each service/daemon is stopped, and then restarting them in the
reverse order. Most important, remember to define a few tests to ensure everything came back up
correctly, including tests of network connectivity if the system was rebooted.

If processes on other systems connect to the AR System server, such as a web server, then you may
want to add a test of those systems to your restart checklist to ensure everything is working properly
afterward. Typically a server restart would be done after hours to minimize the impact on users, so test
thoroughly to avoid coming in the next morning to waiting problems. As a further step, you can consider
automating the process, using available tools appropriate for your platform. But beware, you cannot
automate the tests that everything came up properly.

See also KB 5332, 5113, 5059 that describe some examples, and ways of sequencing the server restart
process.

8. Routine performance checks

You may want to define a process to either spot check performance, or check for performance impacting
problems such as leaving logs enabled. This would be a time to evaluate your archive timeframes, the
number of server processes you have defined on the server, system resources, and so forth. Performing
this check every month or few months might be appropriate.

9. Location of docs and other resources

Collecting documentation and resources in a central location is a good idea so you can reference it,
archive it, and augment it when new resources are available. Collect design documents, requirements
specifications, implementation plans, and any documentation resulting from consulting projects.
10. Maintenance log

A good practice as an administrator is to keep a log of your activities, changes, and updates to the
server. Keeping an electronic copy allows you to archive it in the location suggested above and keyword
search for specific events. Maintenance logs sometimes reveal the root cause of an obscure problem,
which might not be noticed right away. It is a particularly good idea when administration duties are
shared.

11. Special notes or differences

It may make sense to break out a summary of some special circumstances from the maintenance log.
For example, if there a feature has been customized for a different purpose, or if you've been told that a
particular use is unsupported, it may make sense to keep a summary of these details. This might point to
relevant details in an uncommon problem, and be a list of considerations when evaluating an upgrade
plan.

12. Integration points to other systems

For similar reasons, you may want to keep a summary of integrations between AR System applications
and other systems or applications. For example, if there is an integration between a non-ARS Oracle
database, and a particular form in the AR System, a note of what forms, databases, and systems are
involved will be a good reminder of this integration point. The details of a successful integration often
fade with time until some unexpected change steps on it, so this is again a preventative suggestion to
avoid such a problem.
Knowledge Base 6278:
A Primer on reading filter and workflow logs

Diagnosing many different kinds of application problems, questions, and concerns begins with recreating
the behavior with the appropriate logs enabled and sending them to technical support. The reason this is
such a common step in the troubleshooting or investigative process, is that it is deterministic - it tells you
what happened, when, and with what data. Even more importantly, this diagnostic approach can localize
a problem caused by customization, providing insight into functionality, which was done by a third party,
a consultant, or a departed administrator leaving no documentation behind.

But creating logs is not a panacea. There are definite situations when it is more appropriate, things you
can do to make the logs more easy to decipher, and tricks to finding what you need a bit quicker. And in
the case of customization, logging may tell you what happened, without revealing how the intended
functionality is meant to work, and why it differs in some situations. Nor does taking logs definitively
correct the problem in a single step - it may localize a particular problem to a few workflow items you
must look at in Remedy Administrator, or it may just suggest the next step in troubleshooting by
localizing or ruling out potential causes of the behavior.

The paragraphs below are meant to share some of the history, science, and art of reading filter and
workflow logs, and how they are used by technical support to diagnose issues. The topics covered
include:

- A brief genealogy of logs

- Different kinds of logs, and when they are useful
- The basics of reading logs
- Sequence of operations
- Choosing the right text editor to examine logs
- Things to do to make logs more useful
- Using logs to diagnose a few common problems and questions

A brief genealogy of logs

The diagnostics available in the Action Request System have evolved over time, as the features of the
product have grown, and as requested by Technical Support to drive quicker resolution to problems.
Server side logging, including Filter, SQL and API logging have existed since very early versions. Client-
side logging in Remedy User to record the activity of Active Links and Macros was introduced in version
3.2, see KB 6084 on how to enable these logs. Beginning with Action Request System 4.0, these active
link and macro logging could be enabled by selecting an option directly from Remedy User (Choose Tools
a Options, Advanced Tab, and enable the Workflow Logging).

This macro logging was fairly limited, in the sense that it would not report values passed to macros, an
omission which was corrected in Remedy User version 4.03 and later. Beginning with Remedy User
version 4.5, there was also introduced a method to record server-side activity in client side logging,
thereby reducing the amount of information recorded to the logs which, as is discussed later, is quite
helpful in reading log files. (This feature of Remedy User 4.5 is described in the ARS Server
Administrator's Guide Ch 2).

Remedy client side API logging was available in Remedy User 4.01, as documented in KB 3456, but was
subsumed by the Remedy User 4.5x client trace modes (enabling API logging in Remedy User 4.5x)..

On the server side logging capabilities, the Action Request System version 4.5 introduced a more scalable
implementation of server side processing using threads, and with it, logging to track activity of multiple
threads. Prior to this, server logs (filter, API, SQL, etc) would write to different log files with a file
extension indicating which server process handled the activity. For example, after enabling filter logging,
you may see several files on your server: arfilter.log.390621, arfilter.log, arfilter.log.390635, and have to
search all of them for the desired log segment.

As of this writing, ARS 4.52 is the latest version, but later versions may include additional logging
features where appropriate, to facilitate diagnosizing new functionality. Since logging capabilities have
changed, an occasional request from tech support is to use a particular version of Remedy User to enable
logging and reproduce the behavior, as it gives better diagnostic information.

Different kinds of logs, and when they are useful

The most common logs requested are Filter and Workflow (Active Link/Macro) logging. There is a fair
amount of overlap of functionality which can be implemented by Active Links and/or Filters, so a common
request is enable both logs (on the client side, if on Remedy User 4.5) at the same time, and reproduce
the behavior so both can be examined. The most common symptoms reported are either an error
message or a field whose value is changed unexpectedly, and both of these (a Message action, a Set
Fields/Push Fields action, respectively) can be implemented by either an Active Link or Filter. A Run
Process action can also be performed via either an Active Link or Filter, so there are several cases where
these logs are appropriate. Compare the actions available when creating a new Filter, vs. a new Active
Link, and in all of these cases, it'd be appropriate to make concurrent Workflow/Filter logs.

Similarly, escalations can perform these actions, Set Fields, Run Process, etc, but these occur at a time
independent of user activity, and the errors appear in the Arerror.log instead of being presented to the
user. In these cases, an escalation log would be appropriate, typically running the log long enough for
the error or symptoms to occur, and then disabling the log.

SQL logs record the SQL commands sent from the Action Request System Server to the database server,
but does not include the results set returned from the database server. All of the data in forms, and the
definitions of the forms and workflow objects themselves, are stored in the database so SQL logging
generates a great deal of low level information. Interpreting SQL logs requires greater experience, both
with the SQL language and familiarity with the Remedy data dictionary. The Remedy data dictionary is
documented in the later chapters of the "Server Administrator's Guide" (ARS version 4.5), or the "Remedy
Administrator's Guide Volume 2" (ARS version 4.0x) but it's not expected that customers are familiar with
this low level information. SQL logging may be requested by Technical Support in situations of data
corruption, and to troubleshoot searches which return unexpected results, or just sometimes when it'd be
useful to search for a particular value that would only appear in the SQL log.

API logs show Remedy API calls made by a Remedy client (such as Remedy User, Remedy Administrator,
Remedy Import, or any API program such as RappSvc, BBExecd, etc) to the Action Request System
Server. Many but not all API calls have a subsequent SQL call to the database server. Exceptions include
checking user or form information which may be cached in memory on the Action Request System, and
consequently not require a call to the SQL server. The API calls are logged, but not all the parameters to
the API call. API logs record low-level information, which is not expected to be interpreted by customers,
except in cases where they write API programs. The Remedy Programmers Reference describes the
Remedy API calls. Most commonly, API logs will only be requested when written to the same file as the
SQL log, so the SQL calls can be correlated to an API call.

User logging records when users login and logout from the server. This may be useful to troubleshoot
cases where the number of logged in users is different from that expected, but is mostly used
independent of the other logs.

Thread logging is new in ARS 4.5x, to track thread activity. It is used primarily to investigate
performance issues and the load balancing behavior of the ARS Server.
The basics of reading logs

There is a unique prefix at the beginning of each line, indicating what workflow log the line is from. This
is particularly helpful when you enable multiple logs to a single file.

<FLTR> - Filter log

<API > - API log
<SQL > - SQL log
<WFLG> - Workflow log (ARS 4.5x, Active Links and macros)
<ACTL> - Active Links
<MACR> - Macro logging

Server side logs also include a server timestamp and login name of the user initiating the action on each
line. Most of the information that will concern you, however, is further to the right so scroll right to see
the workflow activity.

All logs begin with a line indicate the log has begun, and end with a line indicating it has been disabled.
The looks something like the following:

<FLTR> /* Thu May 17 2001 10:52:03.9740 Demo */

<FLTR> Filter Trace Log -- ON
….
<FLTR> /* Thu May 17 2001 10:52:03.9840 Demo */
<FLTR> Filter Trace Log - OFF

After this, they will include a line indicating the start of some operation. For example, if a user was to
create a new request, there would be a line indicating the beginning of this operation:

Start filter processing -- Operation - CREATE

or
Start active link processing -- Operation - On Window Open

And a line indicating the end of that operation:

Stop filter processing

or
Stop active link processing - On Window Open

Note that in the Active Link/Workflow logs, the end line gives an indication of which operation is ended,
whereas the server side logs just indicate the end of the current operation.

Immediately following the line beginning the operation are two or three lines with additional information
about the operation. Below are three examples. The first is from a filter operation, selecting request # 7
from the AR 4.5 Sampler form. The second is an active link log segment, recorded when opening the
base_ReportTemplate form in a "Create a New" window. The third is another active link log, but on the
operation of selecting an item from a character menu. This sniplet tells us the name of the field with the
attached menu, and indicates the menu is accessed when the form is open in a Create a New window.
As we'll discuss below, information such as the form name, user name, and the Request ID will help us
find the transaction of interest, by keyword searching for these values in the log.

<FLTR> Start filter processing -- Operation - GET

<FLTR> AR 4.5 Sampler - 000000000000007

<ACTL> Start active link processing -- Operation - On Window Open

<ACTL> For Schema - base_ReportTemplate
<ACTL> On screen type - CREATE

<ACTL> Start active link processing -- Operation - On Menu Choice

<ACTL> For Schema - base_ReportTemplate
<ACTL> Linked to field - Primary Audience (600001643)
<ACTL> On screen type - CREATE

In between the start and end line of the operation, you would see each workflow object defined to
execute on this operation is checked, in the sequence defined by their execution orders. In filter logs,
you will also see filters, which are disabled, but in workflow (active link) logs, active links, which are
disabled, are not reported in the logs.

For each workflow object checked, it will report PASSED if the Run If qualification is true, and will show
the If actions performed. If the Run If qualification is not met, the workflow will be reported as FAILED
qualification, and any Else actions will be performed, if any are defined.

Example, from filter log:

Checking CreateMessage (500)

Passed -- perform actions

And from workflow log:

Checking base_Report: Window Close (0)

Passed qualification -- perform if actions

There can be multiple If Actions, or Else Actions, or both, defined for a filter or active link. In the log,
they will be listed in the order defined in the workflow, with the first one being zero. Below, is an
example of an Active Link, which performs three actions, a Set Fields action, a Push Fields action, and a
Change Fields action.

<ACTL> Checking SeveralIFActions (0)

<ACTL> -> Passed qualification -- perform if actions
<ACTL> 0: Set Fields
<ACTL> Assigned To (4) = Demo
<ACTL> Status (7) = 0
<ACTL> 1: Push Fields
<ACTL> To Schema User on Server @
<ACTL> 2: Set Characteristics
<ACTL> For field -- Short Description (8)
<ACTL> Change field to read-only

Two important things to note about the log example above. First, it shows the actual values set, in the
Set Fields action. This is information you cannot get by looking at the Active Link itself, since the value
might be set from a user entered field, or from a different record than expected.

Second note: the partial log above does not tell you from where the data was pulled, or how exactly the
Set Fields action is defined. The Set Fields action may have been hard-coded to the value Demo, or it
could have been defined to be populated from a keyword, a user defined field, or a field on another form.
It also doesn't tell you if a pick list was presented to the user, if multiple entries matched, and the user
selected one with a value of Demo.

Thus, making appropriate log files does not entirely replace looking at the workflow definitions, but it
does help direct your investigation by recording what happened and with what data.

Sequence of operations

Below is an example of a filter log, performing a filter, which has several If Actions. Filter logs are more
involved than Active Link logs, because filter actions occur in multiple phases. (See ARS 4.5 Server
Administrator's Guide, Ch 5, for more information.) In the example, you can see that the Push Fields
action is defined first in the filter, but the action is actually performed after the Set Fields and Message
actions.

Checking FilterWithSeveralActions (500)

--> Passed -- perform actions
0: Push Fields
<deferred to phase 2>
1: Set Fields
Assigned To (4) = Myself
2: Message
Your request has been processed
/* Sun Jun 10 2001 15:14:51.8000 */ End of filter processing (phase 1)
/* Sun Jun 10 2001 15:14:51.8000 */ Restart of filter processing (phase 2)
/* Sun Jun 10 2001 15:14:51.8000 */ 0: Push Fields
<deferred from filter FilterWithSeveralActions>
Email Address (103) = youraddress@yourcompany.com
/* Sun Jun 10 2001 15:14:51.8000 */ Start filter processing -- Operation - SET
User - 000000000000001

The example above is simple, because there is just one filter defined on this form. In real world
situations, the first instance of "0: Push Fields" and the second instance of it, where the action is
performed in Phase 2, may be separated by several pages of other filters executing, and in Phase 2,
there may be several deferred Push Fields actions.

There are a few tricks you can use to make examining the log easier. First, note the line:
<deferred from filter FilterWithSeveralActions>
when the Push Fields action is performed. Another thing you can search for is
0: Push Fields
though this may instead find another filter that has a Push Fields action defined in it.

Choosing the right Text Editor to view log files

All of the log files described above are text-based, and you can use any text editor to view them.

With experience, you'll find there are several features in a text editor that are useful in viewing log files:

Enable/disable word wrap - The server side logs in particular are very long, so if you have word wrap
selected in your viewer it may be hard to see what is going on in the log, as a single line will wrap in your
window to be two or three. Typically, it's preferable to disable word wrap and scroll to the right to see
the other information. The ability to specify a smaller font may also be useful, so you can fit more per
line.
Keyword search both forward and backward - Notepad allows you easily search up or down in the file
from your current field selection, as does MS Word. Wordpad only allows you to search down.

Handling of LF characters - Log files generated on a UNIX server have line feed characters, rather than
CRLF characters, to separate lines. If you copy that log file to a Windows system and view it in Notepad,
the carriage return is not added, and the lines are not separated. Viewing the file in Wordpad makes it
easier to read. If you copy/paste it into MS Word, and then copy/paste it from Word, the CR characters
are added so you can "convert" it to a format easily viewed in other text editors.

How it handles the file in memory - when viewing huge log files, you may want to use Wordpad, or an
editor which handles memory effectively, rather than opening the whole file in memory.

Bookmarks and color coding - Programmers text editors often include the ability to add bookmarks or
define strings to be displayed in a different color. This may be helpful when viewing large log files, by
having Start and End operations in a particular color.

Things to do to make logs more helpful

When making logs to investigate problems or scope changes, there are several things you can do to
make them easier to examine. One is to keep them small, running the logs only long enough to capture
the targeted behavior. Server side logs can become large in a short period of time, so enabling logs after
hours or at a time when other user activity is minimized is preferable. The new ability to enable server-
side logging from Remedy User 4.5 is a big help in this, allowing you to limit server logs to activity
generated by a single user. When doing this, remember to leave the logs on long enough to capture the
behavior, confirming its completion before disabling the logs. A new tool available with ARS 4.5x allows
you to look at log files in real time, to facilitate this. See "Server Administrator's Guide page 2-19 for a
description of using arlogdisplay.exe.

Second, make a note of the request ID, the login name of the user, and the steps performed when the
logs are recording. This will make it much easier to search for the relevant part of the logs.

On ARS Server versions prior to 4.5, separate log files are generated for each server process, using the
RPC numbers. For example, enabling filter logging on a server with MPSO (multiple fast and list servers
enabled), may generate several log files in the same location named: arfilter.log, arfilter.log.390621,
arfilter.log.390635, etc. Remember to search/send all of these log files, as the action may be distributed
between multiple server processes, so examining all the log files is necessary to verify no information is
lost. When convenient, you may want to temporarily disable MPSO (disable all fast and list servers)
before making these server side logs on servers earlier than version 4.5. Typically, that is only practical
on test servers.

Finally, make a note of the behavior, which you observe. Error messages and text may be helpful in
searching the log. If there is a difference in behavior in two different situations, such as different users or
different workstations, make two sets of logs, and note which went with which scenario. This is
information, which is easy to capture at the time that you are making logs, but difficult to remember
afterwards, so it's good practice to jot notes as you make the logs.

Using logs to diagnose a few common problems and questions

A few common scenarios where log files are useful to diagnose or investigate functionality include:

Scenario 1: A field is being set to a particular value on submit, but not sure why, or from where the value
is being pulled.
Using logs: If the update to the field appears as soon as the request is saved, it may be a filter or an
active link, which makes the update, but most likely not an escalation. (if the change only appears
several minutes later, then it very likely could be an escalation.). To investigate: Enable both filter and
Active Link logs, and recreate the behavior. In searching the logs, you may search for the field name
updated, the field id, or the value. Remember that selection lists or option fields will show the value as
the integer position. For example, it'll be set to 0 for the first value.

Scenario 2: An error or warning message appears when you try to make a certain modification to a
request, indicating you must update some other field. You want to determine the source of this error,
and may want to disable it.

Using logs: The message is given by a Message action. As with the previous example, it can be given by
a filter, active link, or escalation, and the most likely candidates are the first two. Enable both filter and
active link logging, recreate the behavior, and search the logs for either the error message, or ":
Message" (that is "colon space Message").

Scenario 3: You want to customize or mimic the behavior of the application in one place, and want to
know what is involved.

Using logs: Since filter and active link logs show you the workflow defined on the operation, even if it
does not pass qualification, it is a more comprehensive view of what functionality may be involved in your
change. If you enable logs and see that there is significant workflow behind the operation, you can
better determine if the change is a simple one, or more of a project.

Ref:
See the "Server Administrator's Guide, Action Request System 4.5" Chapter 2 for additional information
and example of log files.
Knowledge Base 6216:
A checklist for unit testing application customizations

When changing or adding functionality to applications, an important step in the process is to consider the
impact of unexpected data or user behavior, so you can design the appropriate reaction and develop unit
tests which exercise them. That is, put yourself in the role of a quality assurance engineer, and think
about different scenarios and corner cases, which might get you into trouble. In this process, you may
decide some corner cases are too unlikely, or too innocuous to worry about, but your checklist can be a
valuable tool to make sure you didn't forget any that are important. Below are some suggestions.

Unexpected/invalid data - What should happen if a field used in workflow is blank, or the wrong kind of
data, or too long? If the process is user-interactive, you may want to design your workflow to first check
for acceptable values, and give an error to the user otherwise. In other cases, a NULL value is
acceptable, and the qualification of workflow is defined to not execute when the value is NULL.

Users populate fields in other ways - For example, you may define a dropdown character menu to assist
users in selecting a value, but what should happen if they instead type in a value, or select an item from
a pick list (assuming you've added one)?

Users access the data in different ways - Do your changes assume the user will access requests in a
particular way, such as from a shortcut, by querying for a request ID, or opening it from a “control panel”
type form? If so, what happens if they access them in other ways?

Users change data manually - Can users cause problems by changing data manually, after selecting it.
For example, what would happen if a user selected a name from a pick list, then saw that it was
misspelled, and manually updated it? Is that a scenario that you'd rather handle by appropriate user
training, or should you protect against it?

If one match? Multiple matches? No matches? - In cases where data is accessed or updated from other
forms, how should you handle these different scenarios? Your choices are easily configured with the
workflow object, but you may want to add this to your checklist to remember to double check these
settings, or set up unit tests to verify proper behavior.

Affect on different user communities - If your application supports multiple user communities, for
example, users who submit tickets, users who work them, and managers who track metrics, which of
these groups should be affected by your changes? Even if your modifications only target one of these
groups, you may want to include unit tests to verify no adverse behavior to the others.

Affect on different clients? Platforms? Versions? - Should your modifications only apply to certain clients,
or all clients? If you use web clients, such as RemedyWeb, ARWeb, or an application specific web client,
you may want to unit test them. Also, if you have different versions of the clients deployed, you may
want to check the behavior in them. You can use keywords such as $VERSION $ to have workflow
distinguish between different versions.

Impact on email template or automatic submissions via API programs - Should your changes apply to
these cases? If they should, remember to implement changes in filters instead of active links. If they
should not, you can use some flag field set by these mechanisms, to have workflow distinguish and
exclude these non-interactive submissions.

Affect on integrations - If you have integration to other applications, servers, or systems, you may want
to include a test to ensure your changes do not adversely affect it.
Existing data - Test with some existing requests, to see that no problems exist here. Remember that
when you create a new field on an existing form, all the existing records will have a NULL value for this
field. You may want to define a one-time escalation to update those records with a default value, if
appropriate.

What happens when an item is deleted? Archived? Reopened? - This is similar to the case of existing
data. You may want to consider cases like this in your unit testing.

Timing considerations - This is a generic testing approach, but does not come up too much in Remedy
development. You can specify an execution order in workflow, so paying attention to this during
implementation usually prevents this from being an issue.

Frustrated user testing - If there are steps in your application which may take a bit of time, you may
want to add a unit test where you act like a frustrated user, clicking on it repeatedly, just to see what
happens. This is just for your own information, to see if strange behavior or confusing behavior results.
That way you can note it before it is encountered in the field.

Slow system testing - Another good test, if some users have slower systems or small amounts of
memory.

Performance impact - This is usually best handled in design, rather than unit testing. By using efficient
queries, or considering the effect of big queries, you can avoid many performance issues, which might be
hard to track down later. A performance issue is more likely to appear when there is a lot of data, so if
you are making modifications to a system where there is already many requests, you may want to add
selected unit tests to verify your design accounts for the performance concerns.
Knowledge Base 6196:
Suggestions for rolling out application customization

FQ: What practices are recommended to deploy application changes?

How you deploy modifications to your applications can have a significant influence how users receive the
changes and whether new features are fully exercised. Below are some suggestions you may want to
consider to ensure a smooth and successful deployment of application customizations. You may also
want to use this process, perhaps in a more limited sense, when making configuration changes to
applications when the changes would impact existing users.

Note: Investigating, implementing, and testing application customization are natural antecedents to
deployment, but these topics are discussed in other places.

First and foremost, deployment of application changes should be scheduled rather than done on the fly.
The process itself may be quick and easy, but the point of scheduling deployment is in preparation, and
time for contingencies if problems are encountered. As some experienced administrators will attest,
separating the steps of planning and implementation can in itself lead to more thorough preparation, and
fewer hiccups in the process.

Typically, changes are deployed after hours or late in the day so the impact on users will be minimized,
and you'll have plenty of time for unit testing, troubleshooting, or disabling changes if any unforeseen
problems arise. If you have users who work late in the day, announcing the downtime beforehand is
desirable, and can be done at the same time as notifying users of upcoming changes, as discussed in
greater depth below.

At the time that you deploy the changes, you should already have a breakdown of what items need to be
created/migrated to the server to implement the changes. This is information, which should be
discovered in the process of scoping the changes on your development server, and documented in a
design document. For example, you may have a requirement to add a notification to senior management
in a particular case. Your design document may be just a summary of the requirement, relevant
assumptions or notes, and a list of the items that implement the change, including filters, active links,
fields, etc. This list should include the names of the objects to be created/migrated, so that you do not
forget any required components during deployment. Remember to use a naming and numbering
convention for your objects, so you can easily distinguish them from out of box objects.

After making the changes to the application, be sure to unit test the modifications. This is again a case
where having a set checklist ahead of time is preferable. Arguably, the best time to create unit tests is
during the design phase of planning the changes. This is the time when you can think broadly about the
affect of the changes on different users, different aspects and uses of the application, and what can go
wrong with the changes, and work through some scenarios. This preparation ahead of time, again, leads
to more efficient and comprehensive execution during your deployment.

A frequently neglected step in the deployment process is communicating the changes to the affected
users. Ideally, this information should be available prior to deployment or at least, at the time the users
will first encounter the changes. This allows them time to review the changes, and raise questions or
concerns when you can best address them. Without such notification, users may encounter the changes
at differing times, potentially seeing them as glitches in the system, or misinterpreting their intended
purpose.

To facilitate a complete and timely message, it's usually desirable to compose it beforehand rather than
late in the day after deployment. You can always revise it if appropriate prior to posting.
Typical content may include a brief description of the intent of the changes, specific examples of how to
access the new or changed functionality, and maybe a FAQ type summary of anticipated questions. This
may also include a summary of organizational or procedural changes associated with the use of the
application.

Two means of distributing this information to users are common - a broadcast email to user
constituencies, or posting a link or message within a form of the application itself. On the former,
maintaining a current list of users and their email addresses is greatly simplified if the email message can
be distributed from the AR System. This assumes that users do not login as guest accounts and their
preferred notification mechanism is Email.

On the latter approach, of posting a message or link in the application itself, this can be done in Remedy
Administrator by adding a link or message area to a commonly used form, which can be updated with the
latest updates on the application. Advantages of this approach are that the message will be accessible
even by users who do not use email, and the information will be readily accessible from the application
itself, when the information is needed. If an intranet web page is maintained for internal IT information,
adding a link to a web page is an attractive alternative, as is makes the information readily accessible,
can be referenced and updated easily, and can incorporate pictures and graphics to illustrate the
changes.

Finally, after you've completed the deployment, don't forget to update your change log, where you record
your changes to the server. Keeping an accurate change log is important in troubleshooting, planning for
future changes, and for bringing aboard new Remedy Administrators. If you've followed the suggestions
above, the entry into the change log is straightforward. You should include the information from your
design doc on the change and list of components, your standard checklist of unit testing after the
change, and optionally the notification sent out to your users to announce the change. Don't forget to
include the date of the change, and any issues that came up during deployment.
Knowledge Base 6161:
Setting up a dev server, how and why

FQ: How can I set up a development or test server without costing a fortune? Why is it important?

A sound recommendation when customizing or troubleshooting applications is to first test changes on a

development server. In some cases, a development server is not available, making a simple matter much
more risky and difficult to perform. The paragraphs below examine some obvious and not-so-obvious
advantages to having a test server, and some cost considerations in setting one up to meet your needs.

First, on the advantages of keeping a development server. Anyone who has made a simple mistake that
had far-reaching consequences can attest to the necessity of having a test server to prototype changes
and test routine tasks. A good example is writing an escalation or business rule that updates requests or
sends notifications to managers, in which case misstating the qualification can cause a mass update or a
flood of emails to affected parties. Testing it out first would avoid such errors.

Less obvious reasons to have a test server include more productive workdays, efficient troubleshooting,
and better receptivity to changes in the user population. A test server allows you to make changes
during regular business hours without impacting users, and while colleagues and technical support are
available if problems or questions arise. Troubleshooting a problem on a test server is greatly simplified
by having minimal user traffic during logging, and without waiting until after hours. Furthermore, testing
changes on a development server first allows for more successful rollout of new functionality to users,
since it can be fully scoped and implemented, and introduced in a way to minimize confusion. Finally, a
test server is a required component to a healthy administrative routine of planning and preventative
maintenance.

The ideal situation, if cost were no object, would be a test server that was a duplicate of the production
server, with the same data, applications, hardware, and simulated user load. That would provide a very
accurate test bed for changes, as well as investigating performance tuning, upgrade plans, and validating
a backup/restore process.

But for minimal cost, you could instead set up an unlicensed Remedy server on a Windows NT/2000
workstation, either on the administrator's desktop or a reasonable Win-tel system, using an unlicensed
database and the same application versions as installed on the production system. Though this system
would have limitations, it would satisfy the need for the most common use of a test system -
investigating and prototyping changes to application workflow.

At present, the AR System Server and all of the Remedy Applications can be installed and used without
licenses. There are several restrictions with this: No more than three fixed user licenses can be assigned,
no more than 2000 records can be entered per form, and you cannot define multiple server processes.
The impact of this, though, is that you cannot simulate production levels of concurrent user activity, data
load, or use some of the load-balancing and scalability features of the server and applications.

Similarly, several database vendors offer products, which can likewise be used in non-production
environments, with restrictions in performance, data load, and available features. Personal Oracle or SQL
Server Desktop Edition may be an economical avenue to setting up a test server. See your database
vendor license agreement and documentation for details on restrictions, requirements, and limitations.

In practice, a viable test server with the entire suite of ITSM or CRM applications, the database, and all
client tools can be installed on a typical administrator's desktop, if that system is a fairly recent system
running Windows NT/2000, preferably with 512MB or more RAM. Depending on the other purposes of
the system, acceptable response time, and productivity considerations, a better system may be
warranted.
In planning what expense and effort to put into a development system, consider the following:

It's best to match versions of software installed on the production server, except when testing upgrades.
Minimize differences between production and dev environments when possible. To this end, you may
differ from the above mentioned minimum test server, if your server is on a UNIX platform, for example.

If the database version used in production and development servers is the same, you can also use the
system for testing your backup/restore process. This too may warrant a deviation between the minimum
test server approach described above.

If your testing requirements require you to exceed the 3 fixed user write licenses, or 2000 request per
form limitation of the unlicensed Remedy server, you can request temporary evaluation licenses from
Remedy. If you suspect this may be a future use of the test server, you may want to select hardware,
which can support such testing, and investigate whether similar temporary licenses are available for the
database (DBMS) vendor.

Use naming conventions that make it easy to distinguish between production and development servers.
Always have client tools prompt for login, and you may want to use different account information for
production and development servers, to minimize the likelihood of accidentally making changes to the
production server.

If administration responsibilities may be shared with a backup Remedy administrator, then you may want
to put the test server on a separate system from the administrator's desktop or create a Windows login
for the backup administrator so they would have access to the test server during the administrator's
absence.

Remedy Migrator is a separately sold product that facilitates easy movement of data and workflow
between a test and production server, or reporting a list of differences in objects on the two servers. You
may want to go cheap on the dev server hardware and budget for this tool or something similar if
workflow changes will be the primary use of the dev server, or alternatively, invest in a more powerful
test server and choose not to use Remedy Migrator, if backup/restore operations, upgrade planning, and
performance testing are expected to be the primary purpose of the test server.

Remedy,a BMC Software company.Remedy,the Remedy logo and all other Remedy product or service names are registered trademarks or trademarks
of BMC Software,Inc.
© 2003 BMC Software,Inc. All rights reserved.