Application Troubleshooting in a MetaFrame Environment

By Citrix Consulting Services Citrix Systems, Inc.

Notice The information in this publication is subject to change without notice. THIS PUBLICATION IS PROVIDED AS IS WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. CITRIX SYSTEMS, INC. (CITRIX), SHALL NOT BE LIABLE FOR TECHNICAL OR EDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR DIRECT, INCIDENTAL, CONSEQUENTIAL OR ANY OTHER DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS PUBLICATION, EVEN IF CITRIX HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES IN ADVANCE. This publication contains information protected by copyright. Except for internal distribution, no part of this publication may be photocopied or reproduced in any form without prior written consent from Citrix. The exclusive warranty for Citrix products, if any, is stated in the product documentation accompanying such products. Citrix does not warrant products other than its own. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. Copyright 2003 Citrix Systems, Inc., 851 West Cypress Creek Road, Ft. Lauderdale, Florida 33309-2009 U.S.A. All rights reserved.

Version History
1.0 George Prado, Citrix Consulting Services 9/14/2002

Table of Contents
TABLE OF CONTENTS ................................................................................................................................................................ III INTRODUCTION ............................................................................................................................................................................ 1 PROCESS MODEL ........................................................................................................................................................................ 2 1. ESTABLISH A BASELINE ............................................................................................................................................................. 2 2. UNDERSTAND THE SYMPTOMS ................................................................................................................................................... 3 3. REPLICATE THE ISSUE ............................................................................................................................................................... 3 4. CREATE AND VALIDATE THE HYPOTHESES ................................................................................................................................... 4 5. SELECT THE APPROPRIATE TOOL(S)........................................................................................................................................... 4 6. IMPLEMENT THE SOLUTIONS(S) OR WORKAROUND(S)................................................................................................................... 5 COMMON APPLICATION ISSUES................................................................................................................................................ 6 RESOURCE ISSUES ........................................................................................................................................................................ 6 CONFIGURATION ISSUES ................................................................................................................................................................ 8 SECURITY ISSUES .......................................................................................................................................................................... 9 NETWORKING ISSUES ................................................................................................................................................................... 10 METAFRAMES SEAMLESS WINDOW ISSUE ..................................................................................................................................... 10 TOOLS ......................................................................................................................................................................................... 11 CASE STUDY .............................................................................................................................................................................. 13 ACME GDI RESOURCE ISSUE .................................................................................................................................................... 13 Application Issue................................................................................................................................................................... 13 Establish a Baseline ............................................................................................................................................................. 13 Understand the Symptoms ................................................................................................................................................... 13 Replicating the Issue ............................................................................................................................................................ 14 Creating and Validating the Hypotheses............................................................................................................................... 14 Selecting the Appropriate Tools............................................................................................................................................ 14 Implementing the Solution .................................................................................................................................................... 14 CONCLUSION.............................................................................................................................................................................. 15

Application Troubleshooting in a MetaFrame Environment

iii

Application Troubleshooting in a MetaFrame Environment

iv

Introduction
Any applications, certified or uncertified to run on MetaFrame, can potentially encounter issues when installed and utilized in a MetaFrame environment. There are many reasons why applications encounter these issues, but the root causes are usually associated with the following: An improperly configured application or server A poorly implemented application Legacy applications (DOS and 16-bit applications) Application bugs Restrictions in system security

When these issues occur, one or more symptoms may be displayed by the application. The common symptoms are startup failure, application crashes, memory starvation, poor ICA session performance, and poor application performance. Some of these symptoms may be misleading and should be carefully evaluated. The challenge is how to pinpoint the root cause of the issue given the symptoms and clues provided by the troublesome application(s). Identifying the root cause and developing a solution or workaround for the issues described above is not an easy task. However, a process for effective troubleshooting can be defined to guide someone to systematically investigate, analyze, and solve these types of issues. The objective of this white paper is to define a process for systematically resolving application issues within a MetaFrame environment. For this purpose, a problem solving technique called the scientific method was adopted. The scientific method is widely used by scientists to collectively and over time, endeavor to construct an accurate representation of the world. In general this method has the following steps: 1. Identify and understand the problem/question. Ask meaningful questions and state the problem or question in a way that it is conceivably possible to answer it. Gather relevant information to attempt to answer the question. Formulate a hypothesis. Propose a solution or answer to the problem or question. State the hypothesis in a way that can be tested. Test the hypothesis. Test the hypothesis before it is corroborated and given any real validity. There are two ways to do this: conduct an experiment and/or make further observations. Collect and analyze the data. If the hypothesis fails the test, it must be rejected and either abandoned or modified. A modified hypothesis must be tested again. If the hypothesis passes further tests, it is considered to be a corroborated hypothesis, and it can now be published. Make conclusions. The final step is to construct, support, or cast doubt on a theory. A scientific theory is a unifying and self-consistent explanation of fundamental natural processes or phenomena that is totally constructed of corroborated hypotheses.

2. 3. 4.

5.

In troubleshooting application issues in MetaFrame environments, the general steps of the scientific method apply. The issue and its context must be evaluated. The behavior of the application and the associated technical environment must be analyzed to highlight the symptoms of the issue. Based on gathered information, multiple hypotheses can be formulated and test cases can be developed to validate that one or more of the hypotheses are correct. Once hypotheses are proven to be true, workarounds or solutions can be developed to resolve the issue. This paper outlines a detailed process model, based on the scientific method described above, to successfully troubleshoot application issues in a MetaFrame environment. Given this systematic process, the appropriate tools for testing and observing the hypotheses within a MetaFrame environment will be needed. This paper will present a number of lessons learned and software tools that can be used for troubleshooting common application issues. Additionally, a case study is presented to provide a real-world example on how to use the process, techniques, and tools presented in this paper.

Application Troubleshooting in a MetaFrame Environment

Process Model
Troubleshooting applications within a MetaFrame environment requires a systematic approach. The scientific method provides the necessary framework for this approach. Solving the right problem and being productive during the entire troubleshooting process is critical. In many instances, particularly when there is a production issue, the clients bottom line is negatively impacted by it, and the longer it persists, the higher the financial impact to the client. By following a proven process, valuable resources are not wasted by keeping everyone focused on the objectives and tasks at hand, which results in an expedient resolution of the issue. In this section, a detailed process is presented to highlight the important tasks that will assist in the effective troubleshooting of application issues. Below is the highlevel process: 1. Establish a Baseline. Understand the problem. Validate that it is truly a MetaFrame-related issue. Establish the last working state and the current state of the application and the MetaFrame environment. Understand the Symptoms. Observe the symptoms based on the end users perspective, if applicable. Understand the sequence of steps performed by the user or sequence of events that transpired prior to the problem. Replicate the Issue. Replicate the issue in an isolated test environment, if possible. Create and Validate the Hypotheses. Create a list of the probable causes, prioritize the list, and validate. The axiom of Occams Razor can be paraphrased as the simplest solution/explanation is the right one stick with the obvious. Select the Appropriate Tool(s). Find the appropriate tool(s) for testing the hypotheses. Implement the Solution(s) or Workaround(s). Design solutions based on the business requirements and always involve the end user when developing a solution or workaround. Develop the solution(s) and test it. Pilot it in production, if possible or applicable.
U nR e ps t aa td inh e t h e d r lic n e t g s pte S y Im s u o m s

S tta rrtt S a

E E tsat b lis h in g a s a b lis h a B a s e lin e B a s e lin e

2.

U n d e r sa ain g t h e R e p lic t t n d S y m p uoem s Is s t

3. 4.

5. 6.

C e te n d C r r eaat in ga a n d V lid t e t e V aalid aat in gt h h e H y p o tth e s e s H ypo heses

S S lele tcin gh te e ee c t t h A p p rro p rria tte T o o ls A p p o p ia e T o o ls

NOTE: At each stage of the troubleshooting process, the appropriate documentation must be developed and must be updated as new findings are generated. Each task is discussed in detail in the succeeding subsections.
I pp m eent t h I mm lele m n t in gh te e S o lu ttio n s ((s ))// S o lu io n s s W o rrk a rro u n d ((s )) W o ka ound s

1. Establish a Baseline
Understanding the context of the issue is critical to avoid wasting End End resources (time, staff, financial, and technical resources) and to developing the appropriate solution. A solid understanding of the issue will set the direction of how it should be investigated. It is critical to establish the last working state and the current state of the application and MetaFrame environment; that is, did the application work before using a specific configuration or version of the MetaFrame environment, or a specific version of the application. This may establish if the previous changes made to the application or MetaFrame environment introduced the issue. Additionally, by understanding the issue, it may uncover that the issue is not MetaFrame-related. It is also important to understand the severity of the issue as far as the end user is concerned so that the appropriate resources are allocated for addressing the issue.

Application Troubleshooting in a MetaFrame Environment

The following questions should be asked: Does the issue occur in a desktop environment without MetaFrame? If the issue occurs when running the application in a Windows NT 4.0 Workstation or Windows 2000 Professional desktop environment, it is not MetaFrame-related. Does the issue occur in a Terminal Services environment? If the issue occurs in a Windows NT 4.0 TSE or Window 2000 Terminal Services environment without MetaFrame, using RDP clients, it is not MetaFramerelated. Does the issue occur using the base installation of the operating system and MetaFrame? Configuration changes made to a server in addition to the default configuration may prevent applications from working correctly; for example, applying security policies to lockdown the server, changing client drive mappings, etc. Does the issue occur in full desktop window, fixed window, and seamless window modes? The mode in which an application is executed can affect how the application behaves. For example, legacy GUI applications may encounter problems when executed in seamless mode because of the way windows are managed in this mode. However, this problem does not occur in full desktop and fixed window mode. What is the history of changes made to the MetaFrame environment? A specific change or a cumulative change can potentially introduce the issue. These may include hardware configuration changes or upgrades, network configuration changes or upgrades, service packs, hot fixes, application code changes, etc.

If these questions are unanswered, obtain the answers before moving forward. Once the baseline information is available, the context of the MetaFrame environment where the issue is manifesting itself is established. The next step is to attempt to get a detailed understanding of the symptoms of the problem.

2. Understand the Symptoms

Identifying the relevant symptoms and correctly evaluating these symptoms are critical to quickly resolving an issue. Observe the symptoms based on the end users perspective, if applicable. Understand the sequence of steps performed by the user or sequence of events that transpired that is triggering the issue. Monitor the application behavior during the process. Decide if additional probing is needed, which may require using monitoring or debugging tools. An application issue may have one or more symptoms. One or more symptoms may lead towards identifying the root cause. Some symptoms are superficial, which may require additional probing. Some symptoms may not even be related to the issue. The following questions should be asked: What was the user doing before the problem occurred? Did the user observe anything unusual while using the application and the MetaFrame environment before the problem occurred? For example, a sluggish or unresponsive application. Did the user receive any error messages? Can the user replicate the issue?

Once the relevant symptoms have been identified, proceed to the next task and attempt to replicate the issue.

3. Replicate the Issue

Replicating the issue in an isolated test environment is essential. This will ensure that any further investigation will not negatively impact the production environment. In some cases, creating a test environment that can replicate the issue

Application Troubleshooting in a MetaFrame Environment

is not possible because of some discrepancy between the production and test environments, or because setting up a test environment is not feasible. In this situation, create an isolated test environment in production through configuration; for example, use a dedicated MetaFrame server in a production farm for replicating the issue. In some instances, an issue cannot be initially replicated due to various reasons. Make a concerted effort to replicate the issue. Involve subject matter experts if necessary. Further investigation may not be productive unless the issue can be replicated. After the issue has been successfully replicated, start developing a list of probable causes, or hypotheses.

4. Create and Validate the Hypotheses

A hypothesis proposes an answer to an issue. There can be one or more hypotheses that can be created to explain why an issue is occurring. By developing hypotheses and by prioritizing them based on probability, the investigation can be performed in systematic and organized fashion. Some examples of hypotheses are: Memory or resource leak issue System or application configuration issue Network issue DLL issue Security access issue Registry issue

Given the selected list of hypotheses, formulate the test cases that will validate that the hypotheses are right or wrong, consider the appropriate tools for monitoring and debugging, and perform the necessary tests. During the validation process, the root cause may be identified. Once it is found, the next step is to develop the appropriate solution to the issue. In some cases, the root case cannot be identified. In these situations, a workaround may be feasible. NOTE: The test environment must be able to support the testing of the selected hypotheses.

5. Select the Appropriate Tool(s)

Finding and choosing an appropriate tool is of paramount importance to quickly resolving issues. Not only will the appropriate tool provide information to conduct a thorough analysis of the issues symptoms, but also in many cases a tool will be valuable for actually reproducing the issue. For example, an issue may only exhibit itself when the MetaFrame server is under a certain level of user load. The Citrix Server Test Kit, therefore, could be used to generate the necessary load for reproducing the issue. Taking this process a step further, the tool might again be useful for conducting validation once a proposed solution has been developed. When attempting to identify the most appropriate tool for a given issue, the following set of criteria can be used to help narrow the search. Additionally, consider using Internet search engines to identify tools. Simplicity. All things being equal, a simpler tool will be more accurate and easier to use than a more complex tool. Avoid using a tool that provides unnecessary information or introduces unnecessary components into the environment. The more obtrusive the observation, the more likely the thing being observed will be altered by the observation itself. Functionality. Based on the issue being investigated or on the hypothesis being proved, select a tool that actually provides useful information. While a file monitoring utility may ultimately prove useful in troubleshooting database access issues, an ODBC trace will likely provide better information and prove to be a better starting point for analysis.

Application Troubleshooting in a MetaFrame Environment

Speed. Troubleshooting efforts and projects often have aggressive timelines. Choose a tool that is readily available, simple to assemble and implement, and able to quickly provide useful information. Skill Set. All things being equal, a familiar tool will produce results much more quickly than an unfamiliar one. Similarly, a tool based on familiar technologies, platforms, or programming languages will increase productivity. Cost. Use freely available tools whenever possible. For example, unless some extra functionality is needed, the Citrix Server Test Kit can be used to develop user load tests for free. Other load testing tools, however, have licensing fees and may include unnecessary extra features. Also, consider all tools (e.g., SMS, Visual Studio, HP OpenView) that are already licensed by your company and available for your use.

6. Implement the Solutions(s) or Workaround(s)

Selecting the appropriate solution is not always straightforward and it should be carefully evaluated based on the clients requirements. Try to obtain specific requirements before making any recommendations and then develop a list of solutions or workarounds to recommend. Involve the required decision makers in the selection process to ensure that any recommendation is suitable for the environment. Workarounds may be needed when there are no direct solutions to the problem; for example, the issue may require a code change in the application and the source code is not available. Moreover, a short-term solution or workaround may be necessary as a stopgap measure because a long-term solution is not immediately available. In some cases, an acceptable workaround is to do nothing. The solution should be sufficiently validated in a test environment and should be monitored for any unforeseen issues or undesirable effects. Avoid deploying any solution into the production environment without fully testing it. All solutions should be piloted before any wide-scale deployment into production. Also, ensure that a rollback plan is in place in case any unforeseen issues or undesirable effects are introduced by the solution in the production environment. The following are some considerations when making a recommendation: Timeframe. A solution is needed as soon as possible. Budget. The time to develop the solution is cost prohibitive. Administration. Additional responsibility for maintaining the system may be introduced as a result of the recommendation. Performance. A solution may negatively impact the performance of the application or system. Scalability. A solution may not scale to meet the clients requirements. Reliability. A solution may not be reliable enough as far as the client is concerned. Business Drivers. There may be unique circumstances that will restrict the number of options considered.

Application Troubleshooting in a MetaFrame Environment

Common Application Issues

Applications, certified or uncertified for MetaFrame, encounter issues because of the way the applications were developed and the way they were installed and configured in a MetaFrame environment. When application issues occur, they emit one or more of the following symptoms. The application fails to start or initialize That application does not behave correctly or normally The application and/or server crashes Poor application and/or server performance is observed Disconnected ICA sessions occur Slow ICA sessions are observed And more

The above symptoms will require further investigation to identify the root cause of the issue. The challenge is where to start the investigation to get a better understanding of the issue, which will hopefully lead to pinpointing the root cause. There are five technical areas to consider when investigating the probable causes of an issue: resource, configuration, security, networking, MetaFrames seamless windows. The following subsections will discuss in detail each of these considerations.

Resource Issues
Any excessive utilization or misuse of system resources may result to slow performance of applications and servers, and in some cases, may cause applications and servers to crash. One or more of the following resources may be associated with the symptoms of an issue: CPU, memory, disk, network interface, pagefile, files, etc. Poor implementation and insufficient testing of the application particularly within a MetaFrame environment usually introduce the issue and/or leave it uncaught. The following are the common issues: Memory Leaks. This is a common bug of poorly implemented C/C++ programs where dynamic memories are allocated but are not de-allocated when it is not needed anymore. Programs written in other languages such as Java and Visual Basic are not exempted from this issue, particularly when they interface with the operating system, or with legacy and third-party DLLs that are usually written in C/C++. When memory is continuously allocated within a program, the program will eventually run out of memory space to use, which can cause the program to fail. In less severe cases, the application continues to execute but it runs slowly. Memory Starvation. This is a common problem with multi-user operating systems such as Windows 2000 Terminal Services whenever there is not enough physical memory and page file space available, or there are too many applications that are running in the system. This issue usually occurs when there are memory intensive applications that are running and are monopolizing the memory resources of the server. Any instances of applications can be affected because they are contending to use the same memory resource. In many cases, applications will encounter slower performance due to excessive paging. In worst cases, the application may start to fail or not start at all. Resource Limits. The operating system and various applications usually have pre-configured resource limits such as number of files open, number of network connections open, size of the GDI (graphics device interface) heap, etc. These limitations are server- or process-specific. In either case, once a resource limit is reached, applications may start failing unless the applications are implemented to handle such conditions. Applications may also fail to start if the system limits has been reached before an application is launched. Resource Contentions. Applications will compete for one or more shared resources. CPUs, disks, memory, and the registry are resources shared by all running applications within a server. In a multi-user environment, 6

Application Troubleshooting in a MetaFrame Environment

files, shared memory, and network connections are shared by multiple instances of the same application. When resource contention becomes excessive, one or more of these applications will encounter slow performance and, in many situations, will time out while waiting to access a target resource. To pinpoint the root cause, the first step is to identify exactly what resources are utilized excessively or misused, which is tricky because there are cases where multiple resources may be excessively utilized but some of these resources are just side effects. For example, if there are excessive paging and high CPU utilization within a system, the root cause may be insufficient physical memory in the system or the application has a serious memory leak. There are two readily available tools in Windows 2000 to monitor the resource utilization of the system: Task Manager (taskmgr.exe) and Performance (perfmon.msc). These tools are easy to use and are usually the first tools that are used for troubleshooting resource issues. Learn how to use them and how to interpret the data it generates. Figure 2 and 3 below show the list of system resources that Task Manager can monitor.

Figure 1. Task Manager Performance Tab.

Application Troubleshooting in a MetaFrame Environment

Figure 2. Task Manager Resource Monitor. The second step toward pinpointing the root cause of resource contention is to probe into the application by using tracing/logging, instrumentation, and/or debugging tools. Tracing/logging facilities are useful tools for tracking the internal operations of an application. Some application may be configured to activate logging and/or increase the log level. Subsystems that are integrated with the application may also support tracing; for example, ODBCTrace can be used to log the function calls performed by an ODBC-based application. These facilities are run-time features that do not require the applications source code. Instrumentation tools such as memory leak tools and performance profilers are effective tools for resolving memory leak and performance issues. Memory leak tools help identify the specific function within an application program that is causing the leak. Similarly, performance profilers generate reports that list the performance of each function in an application program. When selecting an instrumentation tool, pick the tool that is able to instrument the application executable. For example, Rational PurifyPlus (www.rational.com) offers memory leak and performance analysis capabilities by instrumenting the application executable. In some cases, the only way to pinpoint the root cause is to step through the application processing using a debugger and the applications source code.

Configuration Issues
Configuration issues refer to how the application and MetaFrame server were installed and configured within the MetaFrame environment. The common issues are: Incorrect Installation. All applications that are installed on MetaFrame servers must follow a specific installation process. The server must be in install mode before installing applications. This can be achieved by invoking the change user /install command. After the installation is completed, the server must be restored to execute mode by invoking the change user /execute command. For applications that use the registry to store its configuration data, these applications must be installed using the stated installation process to avoid any run-time issues. Refer to Citrix Knowledgebase article CTX436352 for more details. In some cases, applications may require additional configuration to make it work in a multi-user environment. This configuration can be achieved by developing an application compatibility script for the application. This is a common issue for legacy applications that were not originally designed and/or packaged for a multi-user environment. Refer to the MSDN Library article called Developing Application Compatibility Scripts with Windows NT 4.0 for more details. Additionally, refer to the Citrix Installation Manager Application Compatibility Guide for MetaFrame XP Feature Release 2 for a list of applications that Citrix has experience with. To troubleshoot installation issues, regedit and regedt32 can be used to check that the registry keys 8

Application Troubleshooting in a MetaFrame Environment

associated with the application are correctly specified. In some cases, the filemon and regmon utilities (www.sysinternals.com) can be used to trace how applications access file and registry resources. Service Packs and Hot Fixes. Applications may require specific version levels of each component that it uses, including the operating system and its subsystems, which is usually overlooked. If the appropriate service packs and hot fixes are not applied, the application may fail to function correctly. Always verify the requirements of the application regarding service packs and hot fixes. Dependent Subsystems. Some applications may require specific versions of one of the following, otherwise they will not work at all: Oracle Client, Novell Client, MDAC, COM+ Client, etc. In other cases, these subsystems may have to be specifically configured for the application. Multiple applications installed on a single MetaFrame server may require different versions of these subsystems, which in some cases may not be possible to configure on the same MetaFrame server. Segregation of these applications on separate MetaFrame servers will allow each application to access the appropriate versions of the required subsystems.

Security Issues
Security issues are not application bugs, but are undesirable side effects of the restrictions imposed by a specific security configuration. To overcome such issues, the common approach is to relax the offending security configuration once it is identified. The common issues are: Filesystem and Registry Access. Restrictive NTFS and share permissions can prevent applications from manipulating the files and registry keys it needs, causing them to malfunction. Many applications generate explicit error messages when they encounter such problems. However, some applications do not generate any message at all, which will require additional probing using advanced tools such as regmon and filemon. Group Policies. Group policies determine how programs, network resources, and the operating system behave for users and computers in an organization. Group policies are configured as Group Policy Objects in Active Directory, and affect computers and users by adding, deleting, and changing the values of entries in the Windows 2000 registry. Refer to the Setup and System Administration - Policies and Profiles section of the MSDN Library - Platform SDK Documentation and Planning Distributed Security Setting Uniform Security Policies of the MSDN Library - Windows 2000 Resource Kit for more details. Below is a sample list of group policies that can potentially affect an applications behavior: o o o o o Account. Covers password and account lockout policies. Local Computer. Covers policies to control the rights assigned to user accounts and security groups, enforcing logoff time when logoff hours expire, etc. Systems Services. Covers policies to control the startup mode of Windows 2000 services and the granting of rights to these services. Registry. Covers policies to grant administrators full control over registry keys and their subkeys and to grant read-only permission to other users. File System. Covers policies to configure security for files and folders, and controls the security auditing of files and folders.

Group policies are applied when member computers are started and when users log on to the system. The sequence in which the group policies are applied is based on the Active Directory Organization Unit (OU) structure. The successful application of these policies can greatly influence how an application functions. Refer to Microsofts Q250842 - Troubleshooting Group Policy Application Problems article for details. The common tools used for troubleshooting these issues include secedit (included with Windows 2000 Server), gpresult, and gpotool (both available from the Windows 2000 Resource Kit). Group Privileges. User accounts have associated group privileges. When users log on to the system, their actions within the system are restricted by these group privileges. So when users launch their applications, the users and the applications they are using must comply with the restrictions, otherwise the application may not function correctly or the user may not have access to all required functionality.

Application Troubleshooting in a MetaFrame Environment

Networking Issues
Network-oriented applications such as web, client/server, and distributed database applications may encounter network connection issues such as not being able to establish a connection with a remote component, or experiencing an unstable or unreliable connection. The common issues are: Application Startup. Applications may fail to connect to a remote component causing it to terminate during the startup process or malfunction during run-time. Invalid hostnames, IP addresses, and/or port numbers may cause these issues. Use the appropriate application administration tool or check the application configuration files to verify that needed hostnames, IP addresses, or port numbers are correct. Also, use the ping, tracert, and netstat utilities included with Windows 2000 Server to validate the information. Firewall and Router. Applications may also fail to connect to remote components when firewalls and routers are configured correctly for the applications. Some applications require specific ports to be opened on the firewall. In some cases, the router may not have the information to correctly route a connection request to the applications remote component. The common tools to troubleshoot this issue are the Windows 2000 utilities ping, tracert, netstat, and telnet.

MetaFrames Seamless Window Issue

Many legacy GUI applications (DOS and 16-bit applications) encounter problems when run under seamless mode, but successfully work under full-screen or fixed window mode. In seamless mode, the windows are managed differently; each top-level visible window on the server is presented as a separate top-level window on the client. The ICA client has to manage multiple windows of an ICA session and has to apply common desktop features such as a title bar, minimize and maximize buttons, and exit buttons. Usually, GUI applications that encounter the seamless issue use undocumented windows mode functionality in their programs. Troubleshoot seamless window issues by first determining that the issue is in fact related to executing the application in seamless mode. Once this is confirmed, Citrix Technical Support or Citrix Consulting Services can provide more detailed troubleshooting for seamless window issues.

Application Troubleshooting in a MetaFrame Environment

10

Tools
As previously mentioned, selecting the appropriate tool is the key to quickly and efficiently resolving an issue. Below is a table of some common tools that can be used to diagnose and analyze application issues.

Tool Computer Management System Information DDE Spy/Spy++ Dependency Walker (depends.exe) Dr. Watson Event Viewer GDI Explorer GPO Tool

Purpose Generates a detailed configuration of the system. Provides a graphical view of processes, threads, windows, and window messages. Lists all DLLs used by each process, including version and location. Provides a view of stack traces and unhandled exceptions thrown by Windows applications. Manages the centralized event logs of a Windows system. Monitors the GDI utilization of each process.

Source Included in Windows 2000 Server Microsoft Visual Studio Microsoft Visual Studio

Included in Windows 2000 Server Included in Windows 2000 Server A internal Microsoft tool. Contact Microsoft Consulting to obtain this tool. Windows 2000 Resource Kit

Checks that the client computer has access to an available domain controller.

Gpresult Handle

Displays information about the result that the applied group policies have on the current computer and logged-on user. Lists all the opened files and directories used by each process in the system. Lists all DLLs used by each process including the version and location. Microsofts integrated development environment for developing applications in C/C++, Java, VB, etc. Visual Studio provides the necessary debugging tools for developing applications. Displays protocol statistics and current TCP/IP connections. Allows the sequence of function calls made by an ODBC application to be recorded into a log file. Provides historical data for pre-defined performance metrics.

Windows 2000 Resource Kit http://www.sysinternals.com/

Listdlls

http://www.sysinternals.com/

Microsoft Visual Studio 6.0

http://www.microsoft.com

Netstat ODBC Trace (using ODBC Data Source Administrator) Performance Monitor

Included in Windows 2000 Server Included in Windows 2000 Server Included in Windows 2000 Server

Application Troubleshooting in a MetaFrame Environment

11

Tool Ping Rational PurifyPlus Regmon/Filemon Secedit Task Manager Telnet

Purpose Checks for network connectivity with remote devices. A memory leak tool and performance profiler. Displays information about registry/file access. Used to analyze security settings or create/apply security templates. Tracks counters for memory, processor, disk, etc. in real time. Also provides a list of all processes. A virtual terminal software that can be used to check if a TCP/IP port is open or accessible. A route-tracing utility used to determine the path that an IP packet has taken to reach a destination. Queries and sets information on desktop systems, applications, networks, and other enterprise components through a programming interface. View source code, set breakpoints, view variables, stack traces, and memory. Uses the Visual Studio debug symbol formats.

Source Included in Windows 2000 Server http://www.rational.com/ http://www.sysinternals.com/ Included in Windows 2000 Server Included in Windows 2000 Server Included in Windows 2000 Server

Tracert

Included in Windows 2000 Server

WMI

Included in Windows 2000 Server

Wngdbg

Windows Debugging Tools

Application Troubleshooting in a MetaFrame Environment

12

Case Study
Acme GDI Resource Issue
In this case study, CCS, Microsoft Consulting, and Acme worked together to troubleshoot an application issue. The Acme team is comprised of the various resources from network, application, Windows 2000, and MetaFrame groups. Additional Acme subject matter experts were brought in as needed.

Application Issue
Acme is experiencing an intermittent error when using their Winbroker application within MetaFrame that results in the following error message: "Not enough System Resources to create window." The error is typically followed by an application crash. Acme also observed a second issue where GDIObject resources were continuously allocated when generating reports within Winbroker.

Establish a Baseline
Acme is piloting the latest version of their business critical client/server application called WinBroker. WinBroker was developed using Centura TeamDeveloper and Gupta SQLWindow, and was customized using the SQLWindows proprietary language. The versions of TeamDeveloper and SQLWindows are not certified to run on Windows 2000 Terminal Services or MetaFrame, and the vendor does not support the application on these platforms. There are multiple versions of the WinBroker application utilized at Acme; the latest version seems to have the issue but the older versions has not been verified to have this issue. Acme plans to go live within two weeks with the current MetaFrame environment and the WinBroker application. The MetaFrame environment is comprised of Windows 2000 Server servers with Service Pack 2 and MetaFrame XP Feature Release 1. The applications are published via NFuse 1.61 with Project Columbia. Citrix Secure Gateway is also implemented to connect remote offices that are connecting via the Internet. The issue does not occur on a Windows 2000 desktop. It has not been validated in a Terminal Services environment using RDP clients and on MetaFrames full-desktop or fixed sized windows. Acme has not been able to replicate the application issue consistently.

Understand the Symptoms

After further investigation, it was determined that the two separate issues were in fact related. The continuous allocation of GDIObject resources to the application occurred when dialog windows were open in the application. GDIObjects continued to be allocated when these dialog windows (i.e. Report Progress Bar and Winbroker login windows) were left open and these objects were not released once the dialog was closed. GDIObjects could easily be driven to levels of 6000, 8000, and 9000+ objects in a relatively short period of time while leaving the dialog windows open. Once the GDIObjects allocation reached a threshold, the application would generate the Not enough System Resources to create window error when a user tried to perform a new action within the application. The system resources error appeared to occur sporadically to users, since it depended on when available resources were consumed by the Winbroker application. During the investigation, Microsoft provided a tool called GDI Explorer to perform a more detailed analysis of the GDI resource issue. In one of the sessions, it was observed that when WinBroker opened a dialog window and left it open, the dialog continuously created bitmaps, which caused the continuous allocation of GDI object resources. The developer was consulted about the cause of this behavior. At that time, the cause was not known.

Application Troubleshooting in a MetaFrame Environment

13

Replicating the Issue

Prior to CCS onsite visit, Acme was instructed to make every attempt to replicate the application issue. CCS requested a dedicated test environment so that the investigation can be performed in isolation. All probing tasks performed required the use of Task Manager to monitor the consumption of GDI resources of the WinBroker and associated session. Test servers were utilized to replicate the issue. WinBroker was tested in a Terminal Services environment via an RDP session and the issue did not occur under this configuration. Similarly, WinBroker was tested using MetaFrames full-desktop and fixed window mode and the issue did not occur under the configuration. In both cases, the consumption of the GDI resource remained stable. WinBroker was tested again using seamless windows, and this time, the issue occurred.

Creating and Validating the Hypotheses

The CCS and Acme teams met to discuss the current findings, and to develop the following next steps: 1. Investigate if the application code has a memory leak; if it is actually allocating bitmaps for dialog windows. The developer reviewed the code, and the application logic does not continuously allocate bitmaps. Based on the information about the symptoms, the developer searched a number of Internet-based Centura/Gupta newsgroups for any similar cases related to WinBroker. He found an article stating a fix that requires passing an option (WM_QUERYDRAGICON) when creating dialog windows within WinBroker. This proposed fix was applied and the change seemed to fix the issue. Investigate if there are any seamless window related patches that can be applied to the published application. The CCS consultants were familiar with the seamless window issue and consulted their internal resources for a possible fix. CCS identified the Citrix articles that guided them on how to apply the necessary seamless exceptions to the registry. The changes to the registry were scheduled but not completed. Investigate if there are any application, Microsoft and MetaFrame hot fixes that can fix the issue. No hot fixes were identified. Investigate if the application is using the wrong DLL versions. CCS worked with Acme to verify that all subcomponents of the application had the correct versions. The appropriate tests were performed without any positive results. Investigate if adjusting the ICA settings (resolution, color, session graphic buffer, etc.) can resolve the issue. The appropriate tests were performed without any positive results.

2.

3. 4.

5.

After further testing, Item #1 fixed the issue. It was not necessary to pursue the other investigation tasks.

Selecting the Appropriate Tools

The tools used in the investigation were mainly monitoring tools that were used when replicating the application issue. Task Manager was the first tool used to monitor the GDI usage; perfmon was not used because it is not capable of monitoring GDI resources. However, during the investigation, more detailed information about the GDI consumption of the application was needed. Microsoft provided a more advanced GDI monitoring tool called GDI explorer.

Implementing the Solution

The solution was deployed to the pilot environment for further testing by live users. This occurred for a couple of days to ensure that it was fully validated. The required changes were applied to the rest of the application code. CCS recommended to Acme that a comprehensive regression test of the application be performed before the application was moved to production. Application Troubleshooting in a MetaFrame Environment 14

Conclusion
Troubleshooting applications in MetaFrame is very similar to troubleshooting any software application; that is, a clear understanding of the issue and its symptoms are essential before pursuing any other troubleshooting activities. This will set the stage on how to investigate the issue, find the root cause, and eventually develop the appropriate solution. It is critical to use a systematic process during the troubleshooting effort to ensure that the issue can be evaluated objectively. Managing the expectation of the stakeholders is critical. Using a systematic process will avoid wasting time and resources. In a highpressure situation, a standard, documented troubleshooting process is indispensable. The troubleshooting process presented in this paper is a roadmap. It sets the direction on how to reach a destination; in this case, a solution to an issue. It suggests some options on how to arrive at a solution. It provides a number of tips to expedite the troubleshooting process. However, there is no silver bullet. Each issue is unique and the technical environment and business requirements influence the development of an acceptable solution.

Application Troubleshooting in a MetaFrame Environment

15

851 West Cypress Creek Road

Fort Lauderdale, FL 33309

954-267-3000

http://www.citrix.com

Copyright 2003 Citrix Systems, Inc. All rights reserved. Citrix, WinFrame and ICA are registered trademarks, and MultiWin and MetaFrame are trademarks of Citrix Systems, Inc. All other products and services are trademarks or service marks of their respective companies. Technical specifications and availability are subject to change without prior notice.