ManageEngine Desktop Central Patch

Management SOP
By Softcell Technologies Limited

SUBMITED TO
Sterlite
Approvals

Issued by: Suraj Pillai (srpillai@softcell.com)

Reviewed by: Sameer Panadare (sameerp@softcell.com)

Issued To: Sterlite

Copy No. 1

Document Version: 1.1

Date: Sept 25th, 2017

Softcell Technologies Limited.

401-402, Mayfair Towers-I, 28,Mumbai-Pune Road, Wakdewadi, Pune-411005

Phone: +91.20.66006700, Fax: +91.20.66006701, URL: http://softcell.co.in

India (Bangalore, Chennai, Hyderabad, Mumbai, New Delhi, Pune)

Notices

Copyright @ 2/11/19 Softcell Technologies Limited

All Rights Reserved.

The information contained in this document is confidential. It is suitable only for use for its intended
purpose and may not be disclosed to third parties or modified/changed without express written
agreement of Softcell Technologies Ltd.

Softcell believes the information in this publication is accurate as of its publication date; such information
is subject to change without notice.

Softcell conducts its business in a manner that conserves the environment and protects the safety and
health of its employees, customers, and the community.

Softcell, the Softcell Technologies logo, are trademarks of Softcell Technologies Limited.

Notices

© 2016 Softcell Technologies Limited

This document is the property of and is proprietary to Softcell. It is not to be disclosed in whole or in part
without express written authorisation of Softcell. It shall not be duplicated nor used, in whole or in part,
for any purpose other than to evaluate Softcell’s Proposal and should be returned upon request.
Ref:
Date: 25th Sept 2017

To,

Sterlite,

Pune

Kind Attn: Mr. Aazam Razvi

Sub: ManageEngine Desktop Central Patch Management SOP

Dear Sir,

This refers to our discussions regarding your requirement of Standard Operation Procedure for Patch
Management, using ManageEngine Desktop Central for your company. Should you need more
information on any of the points in the proposal please contact us and we will be pleased to provide you
with adequate inputs to support you during the decision process.

We look forward to partnering with you and contributing to the success of this important project.

Thanking you,

Assem Dubey,

Strategic Engagement Manager

Softcell Technologies Limited, Pune

CONTENT

1. STANDARD OPERATION PROCEDURE OF PATCH MANAGEMENT

2. OVERVIEW
3. DESIGN RECOMMENDATIONS
4. PATCH DEPLOYMENT PROCESS
5. TESTING PROCESS
6. SOFTWARE UPDATE ROLLOUT STRATEGIES
7. DISTASTER RECOVERY PLANNING
8. REBOOTING
9. REMEDIATION
10. SCANNING AND REPORTING
 STANDARD OPERATION PROCEDURE OF PATCH MANAGEMENT

1. ZOHO downloads Patch, assessment for patch authenticity and testing for functional correctness
is carried out at this site. The final analysis and data are correlated to obtain a consolidated
vulnerability database which serves as a baseline for vulnerability assessment in the enterprise.
The modified vulnerability database is then published to the Central Patch Repository for further
use.

2. The Desktop Central Server is located at the enterprise (customer site) and subscribes to the
Central Patch repository, to periodically download the vulnerability database. It scans the systems
in the enterprise network, checks for missing and available patches against the comprehensive
vulnerability database, downloads and deploys missing patches and service packs, generates
reports to effectively manage the patch management process in your enterprise.

3. Manage Engine releases patches mostly for all OEMs, so it is advisable to enlist the standard
software used in the firm, so as to download only the useful patches.

4. All patches, i.e. Critical, Important, Moderate, Low, and Unrated (Optional), should be
downloaded, because, few patches installs only if dependencies are installed.

5. Patches are released on 2nd Tuesday Monthly, these patches should be tested and deployed within
4rth week of the Month.

6. Released patches should be analyzed and prioritized before testing them, e.g. Patches for
Windows OS with severity as Critical should carry more priority than that of Critical patch for
Microsoft Office or Adobe Reader. Critical vulnerabilities that have published exploit code should
be given the highest severity weighting.

7. Manage Engine works in a Domain Infrastructure, so all machine should be in Domain to get
patched successfully.

8. 4. For Workgroup Infrastructure, Manage Engine facilitates with the provision of grouping the
workgroup machines assigning credentials for scanning and patch deployment. So credentials
should be well managed.

9. Various Enterprise level patch testing scenarios:

 Lab Testing / Smoke Testing – VM setup of similar flavors of OS mostly used.
 Pilot Testing – Test groups would include machine of users in different sites.

10. Patches should be tested with following test cases:

 Installation Tests— to validate that the patch installs without error, and that any launch
conditions contained in Windows Installer patches are working properly.
  Verification Tests— to verify that shortcuts, help files, and file associations set or modified by
the patch are working properly.
 Execution Tests—to verify whether the files and registry keys created or modified by the patch
can be read and updated when the application is executed by typical users who do not have
administrator-level privileges.
 Standard Tests—to verify that the installation of a patch does not negatively impact the ability
to execute another application found on the desktop or the ability to connect to a URL,
network share, or database.
 Rollback Tests—to verify the safe method of uninstalling the patch and/or restoring the target
computer to the pre-patch state in the event of a conflict.

Understand a patch’s impact on the environment before deployment.

11. Once tested, patches should be deployed to all the vulnerable machines.

12. Patch Scanning techniques should be used, populating reports so as to identify the status of the
patch deployment process.
 Patch Scanning – Patch Scanning helps to determine the healthy and vulnerable machines. It
provides details like:
 Healthy Machines
 Vulnerable Machines
 Missing updates – Critical, Important, moderate, low.
 Installed updates.

We can export Report for all scans and these can also be scheduled to be run and available to
export as xlsx, csv and pdf formats.

13. Inventory Scan and Patch Scan should be planned and schedule as per the maximum availability of
the machine in the network.
14. Patching over Internet, this will require port forwarding in the network gateway, by virtue of which
client agents will connect to Distribution server or Desktop Central.
15. Recovery steps should be followed, in case if required:
 Uninstall the patch through Add or Remove Programs—this method is not scalable for
widespread problems.
 Uninstall the patch through a command line—this method can be replicated to all affected
computers through AD or Desktop Central.
 Restore the desktop to an official corporate image—this is a much more drastic method and
will cause productivity losses.

16. Patch Management Solution provides a wide variety of options for reboot scheduling. Reboot is
very important for a machine to be shown as healthy, once patch is installed, so Reboot should be
a regular practice after patching.

17. Compliance report should be scheduled via automated email reporting to the IT Administrator.
OVERVIEW

Patch Management is a practice designed to proactively prevent the exploitation of vulnerabilities on IT

devices. The expected result is to reduce the time and money spent dealing with vulnerabilities and their
exploitation. Taking a proactive approach to patch management will reduce or eliminate the potential for
exploitation and involve considerably less time and effort than responding after vulnerability has been
exploited.

This document details industry best practices for patch management and the Desktop Central best
practices.

Patch Repository: zohocorp

Patch download, assessment for patch authenticity and testing for functional correctness is also carried
out at this site. The final analysis and data are correlated to obtain a consolidated vulnerability database
which serves as a baseline for vulnerability assessment in the enterprise. The modified vulnerability
database is then published to the Central Patch Repository for further use. The whole process of
information gathering, patch analysis and publishing the latest vulnerability database occurs periodically.
The Central Patch Repository is a portal in the Zoho Corp. site, which hosts the latest vulnerability
database that has been published after a thorough analysis. This database is exposed for download by the
Desktop Central server situated in the customer site, and provides information required for patch
scanning and installation.

The Desktop Central Server is located at the enterprise (customer site) and subscribes to the Central Patch
repository, to periodically download the vulnerability database. It scans the systems in the enterprise
network, checks for missing and available patches against the comprehensive vulnerability database,
downloads and deploys missing patches and service packs, generates reports to effectively manage the
patch management process in your enterprise.

DESIGN RECOMMENDATIONS

Successful Patch Management requires a robust and systematic process. This process, the Patch
Management Lifecycle, involves a number of key steps:

Defining Service Levels

When defining service levels, it is first necessary to understand that not all patches are created to resolve
security vulnerabilities and that the severity of the vulnerability will vary. Service levels should be defined
as a combination of a time frame based upon Severity Level to reach the minimum compliance level. For
example:

 The Service Level for remediation of applicable Critical severity vulnerabilities is 3 business days
at 87 percent compliance.

Microsoft’s Severity Rating System

Microsoft’s vulnerability rating definition is provided below. Microsoft’s severity rating system provides a
single rating for each vulnerability. The definitions of the ratings are:

Rating Definition
Critical A vulnerability whose exploitation could allow the propagation of an Internet worm without
user action.
Important A vulnerability whose exploitation could result in compromise of the confidentiality, integrity,
or availability of user’s data, or of the integrity or availability of processing resources.
Moderate Exploitability is mitigated to a significant degree by factors such as default configuration,
auditing, or difficulty of exploitation.
Low A vulnerability whose exploitation is extremely difficult, or whose impact is minimal.

(Reference: http://www.microsoft.com/technet/security/bulletin/rating.mspx .)
Note:

1. Define the company standard software list which requires to be patched. For example, if the
vulnerability only exists in MS Outlook, and the organization is a Lotus Notes shop, then the risk
of exposure is reduced for that particular vulnerability.
2. Critical vulnerabilities that have published exploit code should be given the highest severity
weighting.
3. Most organizations with an automated patch distribution mechanism establish a short time frame
(the unofficial average is 48 hours to 2 week) for the testing and distribution of critical patches.
4. Patches that resolve Important to Moderate severity vulnerabilities are commonly given
additional time for testing and deployment.
5. The management of Low severity vulnerabilities varies frequently from company to company.
Some organizations do not track Low severity items, while others treat them as equivalent to
Moderate severity for purposes of testing and deployment. The tradeoff is between the minor
increase in risk and the potential loss of software functionality.

Compliance Levels

A compliance level refers to the percentage of computer devices that have been successfully patched or
otherwise remediated such that they are no longer vulnerable. Setting a reasonable goal for compliance
levels is often a difficult concept. At first glance, a completely patched environment (100%) would appear
to be a realistic goal.

However, this does not take into account the reality of:

 Users that own multiple computers (which aren’t always connected and powered on).
 Traveling employees with notebook computers that don’t log into the corporate VPN every night.
 Computers being repaired or replaced by a hardware vendor.
 Contractor’s computers that may or may not even still exist on the corporate network, yet still
show up on recent network inventory reports.
 Failed or over saturated WAN connections.
 Computers registered in Active Directory that have been repurposed and/or re-imaged.
 Virtual computer sessions that remain powered off for weeks or months at a time.

The most commonly observed compliance levels are between 85 and 97 percent. Some organizations go
the additional step of setting a multi-phased compliance goal, such as 91 percent within 7 hours, 97
percent within four weeks, and 99 percent within 6 months.
PATCH DEPLOYMENT PROCESS

The cycle begins with the release of security patches on the second Tuesday of each month. The
Operations team receives the list of security and non-security updates and creates the update installer
package. The Operations team installs the updated packages to their Configuration Manager
infrastructure first. That helps validate the update package and ensure that the servers are ready to
deploy the patch to smoke test and production servers.

Patch Assessment and Scanning

Desktop Central periodically scans the systems in your network to assess the patch needs. Using a
comprehensive database consolidated from Microsoft's and other bulletins, the scanning mechanism
checks for the existence and state of the patches by performing file version checks, registry checks and
checksums. The vulnerability database is periodically updated with the latest information on patches,
from the Central Patch Repository. The scanning logic automatically determines which updates are
needed on each client system, taking into account the operating system, application, and update
dependencies.

On successful completion of an assessment, the results of each assessment are returned and stored in
the server database. The scan results can be viewed from the web-console.

Patch Download and Deployment

On selecting the patches to be deployed, you can trigger a download or a deploy request. At first the
selected patches are downloaded from the internet and stored in a particular location in the Desktop
Central server. Then they are pushed to the target machines remotely, after which they are installed
sequentially.

TESTING PROCESS

The patch testing process is composed of two stages: Preparation and Testing
Preparation

The goal of the preparation phase is to:

 Understand a patch’s impact on the environment before deployment.

 Identify possible file version conflicts between the contents of a given patch and pre-existing
software in the environment.
 Identify which applications might be impacted because of their dependence upon updated
system-level files.

Having this knowledge will greatly reduce the number of test cases to be performed before a patch is
considered to be safe for deployment. It will also help to define the current vulnerability of the
environment.

Testing

Test cases are the documentation of the procedures, targets, and expected results for each individual test
to be performed. When building a list of test cases for a patch, include each of the specified test-case
types.

Types of tests cases

 Installation Tests—To validate that the patch installs without error, and that any launch
conditions contained in Windows Installer patches are working properly.
 Verification Tests—To verify that shortcuts, help files, and file associations set or modified by the
patch are working properly.
 Execution Tests—To verify whether the files and registry keys created or modified by the patch
can be read and updated when the application is executed by typical users who do not have
administrator-level privileges.
 Standard Tests—To verify that the installation of a patch does not negatively impact the ability to
execute another application found on the desktop or the ability to connect to a URL, network
share, or database.
 Rollback Tests—To verify the safe method of uninstalling the patch and/or restoring the target
computer to the pre-patch state in the event of a conflict.

Lab Testing

Initial patch testing should always be performed on non-production computers. To facilitate adequate
testing of automatically deployed patches, organizations should use standardized configurations for IT
devices as much as possible. A popular technique is to leverage virtual machine technology that provides
a rollback capability to facilitate rapid testing and retesting against common workstation and server
configurations.

Primary testing candidates are those configurations that exist in the production environment, such as:

 Each workstation OS per service pack level (Windows 98, Windows 2000 SP4, Windows XP gold,
Windows XP SP2, and so on)
 o Each of the above with core workstation applications installed (MS Office, WinZip,
antivirus client, Acrobat, and so on)
o Each of the above with any custom/proprietary applications in the environment
 Each Server OS per service pack level (Windows 2000 Server SP4, Windows 2003, Windows
20003 SP1, and so on)
o Each of the above with core server applications (IIS, primary Application Server software,
Antivirus client)
o Each of the above with any custom/proprietary applications in the environment

More focused testing should be performed on replicas of business critical servers (customer facing
servers, HR and ERP applications, mail servers, database servers, and so on)

Pilot Testing

Roll out the patches to one or more pilot groups. As previously discussed, pilot groups should focus on
commonly used configurations in the production environment.

Good pilot groups are those that are readily accessible by IT staff in the event of a failure, computers
assigned to the more technically savvy users, and computers that are the least likely to impact the
primary business functions of the organization in the event of failure.

After comparing the differences between the pilot testing and the earlier testing phases, develop a
rollout plan that addresses any observed issues. The recommended technique of a multi-phased rollout is
discussed under the Software Update rollout strategies section of this document.

SOFTWARE UPDATE ROLLOUT STRATEGIES

A successful rollout strategy should include multiple phases. A phased approach provides checkpoints at
which problems can be identified and constrained. Below is the foundation for any desktop rollout
strategy.

Phase 0—Smoke testing (1 Week)

 Only target personal test computers.

 Validate proper command line parameters for the patches.
o Patch installed correctly.
o Proper level of user interaction.
 Completely silent?
 DOS window flashing open and then closed?
 User response required?
o Reboot required?
 Check for blue screens (fatal OS errors).
 Check for obvious performance issues.
 Check for dependency prompting (such as MS Office source files).
 Core applications still execute as expected (MS Word, WinZip, IE, and so on).
 Intranet portal accessible?
 Schedule policy should be ASAP.
Phase 1—Pilot groups (2–5% of total environment) (1st – 2nd Week)

 Should be easily recoverable computers.

 Target users that are more tolerant of IT related problems.
o IT staff
o Non-business critical computers.
 Some customers will also target the secondary computers assigned to application owners.
 Other customers maintain a pilot lab in which each primary business application is installed.
Application owners are then responsible for approving the patchs after performing unit and
integration testing.
 Schedule policy should be ASAP.

Phase 2—Majority rollout (40–75% of total environment) (2nd Week)

 Breakout the rollout targets by utilizing collections representing major sections of the
environment, such as:
o By AD organizational unit (OU).
o By type/purpose of computer.
o By business unit.
 In this case, some customers assign responsibility to business unit leaders for
managing which computers are in their respective collections.
 This approach also allows each business unit to select a default installation time
and rebooting behavior.
 Schedule the patch to install several hours in the future to allow sufficient time for it to be cached
on the clients.
 Schedule should match what makes sense in your environment.
o Do users leave computers on the network at night?
 If not, then you should probably schedule installation during working hours or
enable Wake on LAN functionality.
o Do you have 24/7 operations?
 Establish a maintenance window.
 See the Rebooting section below for dealing with shared, multiple-shift
computers.

Phase 3—Final rollout (Remainder of environment 3rd Week)

Invariably, each customer will have computers that require special attention during Patch Management
operations. In many of these cases, the impact of a patch-related fault is considered a greater risk than
that of the vulnerability being exploited. After observing the results of the prior phases, a determination
can be made about automating the patch rollout, manually installing the patch, removing the vulnerable
applications, configuring a personal firewall, or removing the at-risk computer from the network.

Sample computers include:

 Factory floor automation.

 Vendor maintained systems.
 Traveling users with VPN access.
Financial staff computers during end-of-quarter or end-of-year operations.

DISTASTER RECOVERY PLANNING

In the context of patch management, disaster recovery is limited to resolving problems caused by the
installation of a patch. This section provides the most common techniques. As previously discussed, part
of proper patch testing includes verifying that a patch can be safely removed, and having a viable fallback
plan in the event the patch can’t be correctly uninstalled.

Traditional desktop recovery

1. Uninstall the patch through Add or Remove Programs—this method is not scalable for
widespread problems.
2. Uninstall the patch through a command line—this method can be replicated to all affected
computers through AD or Desktop Central.
3. Restore the desktop to an official corporate image—this is a much more drastic method and will
cause productivity losses.

Traditional server recovery

1. Backup the server prior to deploying the patch.

2. Uninstall the patch through Add or Remove Programs—this method is not scalable for
widespread problems.
3. Uninstall the patch through a command line.
4. Restore the server from a backup.
REBOOTING

Patch Management Solution provides a wide variety of options for reboot scheduling. This section
discusses commonly used configurations and the thought process behind the configuration.

Reboot Options

The following basic reboot options are available

 Do not reboot after installing patches.

 Reboot immediately after installing patches.
o And if necessary, you may allow multiple reboots during the patch window (this is rarely
needed with MS patches released in the last two years).
o Reboot at system startup.
 Force Reboot

Reboot Notification

The following notification options are available:

 Never
 Once (after applying the patch).
 On a schedule (with repeating warning messages if desired).
 Warn the user about a pending reboot in X time.
 User can defer reboot for X time.

Note: Reboot is very important for a machine to be shown as healthy, so Reboot should be a regular
practice after patching.

REMEDIATION

In the context of patch management, remediation refers to the steps taken when a fault occurs as result
of the patch process.

Patch Installation Causes Application or Operating System Failure

In this situation, the patch must be removed until the application or hardware vendor can supply a
corresponding fix. To prevent re-installation of the patch, exclusion for the affected computers should be
added to the corresponding collection for the patch policy.
Patch Fails to Install

In this situation, the patch was scheduled to be installed, but it still isn’t present on one or more
computers after sufficient time has elapsed.

Symptoms

 Patch not listed in Add or Remove programs (not a consistent detection method).
 Computer continues to show up on the Compliance by Bulletin report as being out of compliance.
 Secondary vulnerability scanner indicates patch is not installed.

Diagnostic techniques

 Review the software update summary data by opening the Resource Manager on the problem
computer > Summarytab > Software Update Summary> Return Codecolumn for the
corresponding software delivery task.
o Lookup the returned error code.
 Google for “msi error code 12345”.
 Google for “error code 12345”.
o Review the TechNet article for the bulletin.

 Manually execute the patch (without switches) on a problem computer.

o Identify and resolve the computer-specific problem. Replicate the fix through Software
Delivery Solution or through a Deployment Server job to other affected computers
o Some patches trigger the application to prompt for MSI source files—invalid MSI source
paths for MS Office installations can be fixed through a registry key change.
o Check for any running instances of msiexec.exe, setup.exe, or install.exe. This is a strong
indication of a hung installation job from another activity.
o A prior package installation may have set the Windows installer reboot flag. Additional
production installations will refuse to execute until the computer has been rebooted.
o On some occasions, the manual installation process will succeed without providing an
obvious reason for the previous failures. This is generally caused by the patch requiring
the target application to be closed prior to installation. Because of this situation, the
default mode for the software update agent is to retry installation three times.
 In some environments, such as those where users infrequently close MS office
applications, it may be necessary to increase the number of attempts to four or
five.
SCANNING AND REPORTING

1. Asset Scanning – Asset Scanning helps to maintain an updated IT inventory in the network. It
provides details like:

 Total Machines in the network

 Machines with Agents
 Machine Available in network
 Last scanned status.

Asset scanning should be scheduled in 1st and 5th day of the week. Scheduling interval would
depend on inventory change frequency.

2. Patch Scanning – Patch Scanning helps to determine the healthy and vulnerable machines. It
provides details like:

 Healthy Machines
 Vulnerable Machines
 Missing updates – Critical, Important, moderate, low.
 Installed updates.

We can export Report for all scans and these can also be scheduled to be run and available to export as
xlsx, csv and pdf formats.