What Patching Policy Should I Adopt For My Red Hat Enterprise Linux Servers? - Red H...

Page 1 of 3

What Patching Policy Should I Adopt For My Red Hat Enterprise Linux Servers?
( Updated November 5 2013 at 2:21 AM

0 Issue
In large estates of RHEL servers, where there is a restrictive change control policy, it is possible that systems may not receive errata in a timely fashion. In some cases this can lead to unexpected reboot events for a problem that has already been fixed in errata. What patch strategy should I use for my servers?

Environment
Red Hat Enterprise Linux, All Versions

, Resolution
Red Hat does not recommend a particular patching policy for individual installations. Every Systems Administrator must evaluate the best way to maintain their estate of installations. We recognise that many large environments have a change control process that must be strictly adhered to and we offer the following as basic guidance for those who are developing a policy for their own organisations.

Comments
There are essentially two approaches to system updates, proactive and reactive. These two approaches can again be subdivided into active and passive policies. A Proactive, Active policy dictates that the systems administration team are aware of every erratum released to the repositories and that they review the contents to determine if each erratum is relevant to the system. A Proactive, Passive policy requires that at an appropriate schedule, all available errata will be applied to a system. This schedule would be dictated by the availability of the systems in question and the testing policy required before pushing an errata to production. A Reactive, Active policy is where a problem is found on the system and a specific errata is applied to remedy it. A Reactive, Passive policy is where a problem is found on the system and it is then updated with every available errata at that point. While it is a little belaboured to cover these differences, it is important to do so, because there may be the necessity for different groups of systems to be covered under different policies, depending upon their role in the organisation. For example, a firewalled mail server accessible only via SMTP and IMAP/POP is only going to need to receive errata against its core applications, remote attacks, or if a problem is found on the system. The relative merits of each policy are utterly dependent on the individual systems and the availability required of them, not to forget the workload and associated cost of performing updates. For example, if you were running a number of hardware and virtual servers that provide services to a small team, then a Proactive, Active policy is easily maintainable. This is also maintainable when you have a limited number of standard builds, even across a large installation. With a Proactive, Active policy on Hardware, a Proactive, Passive policy towards virtual machines that are not mission critical is another workable approach. This is because with a virtual machine it is easier to take a snapshot and roll back if an update causes more problems than it fixes - it is rare that this sort of thing happens, but it is a reality which we must account for in our approach to systems administration. A Reactive Passive approach would likely be done at every point release for Red Hat Enterprise Linux systems and every Fedora Release minus one. So when Red Hat Enterprise Linux 6.2 is released, the Red Hat Enterprise Linux 6 VMs will be updated after taking a snapshot, and when Fedora 16 is released, the F14 VMs will move to F15 + Errata. When you are dealing with high uptime objectives, a Proactive, Active approach, while time consuming may be the best way to ensure success. This would need to be managed as part of a regular cycle of testing and promotion to live. In this sort of environment you may also wish to consider Red Hat Network Satellite (http://www.redhat.com/red_hat_network/) .

https://access.redhat.com/site/solutions/64094

11/25/2013

What Patching Policy Should I Adopt For My Red Hat Enterprise Linux Servers? - Red H... Page 2 of 3

Many customers find that they wish to standardise upon a particular point release of Red Hat Enterprise Linux and do not move forwards with errata. In these circumstances it is strongly recommended that you investigate Extended Update Support (/knowledge/node/22763) (EUS), as this errata stream will allow that to happen. This approach is often the only real way to maintain an estate at a point release, especially where there are third party applications and strong change control proctices in place. It should be noted that a point release of Red Hat Enterprise Linux is simply the major release and the available errata at a point in time. In most cases following the latest errata after testing in your environment will be an acceptable approach. Some systems seem to just run forever with absolutely no issues, but for some reason the highly visible (and frequently the busiest and most used) systems are the ones seen to fail. Whichever patching policy you choose to go with, we urge you to consider your kdump policy at the same time. The inclusion of kdump in an article about patching at first appears odd, but he importance of getting a vmcore from a panicked system should not be underestimated. It can mean the difference between one unplanned outage followed by application of errata or several events on the system while we gather data. In the event you find a new issue, a vmcore will allow us to produce an erratum quicker than without. If you have a Proactive, Active policy it will allow you to see if an erratum has been missed, or if you have found a new issue, which in turn makes it easier for Red Hat to respond to you. A central kdump server is often not possible due to firewall restrictions, but one kdump server, accessible via ssh from every production node in the estate could save you a fortune in lost productivity. Kdump and the configuration of it are extensively discussed elsewhere in the kbase, simply search for kdump above. Here are two real world examples of patching strategies. Customer 1 Takes a Reactive, Active approach to most of their systems, but takes a Proactive, Active approach to all security errata. Takes the Red Hat build and adds configuration directives to disable unneeded services. Tends to take every odd point release (5.1, 5.3, 5.5 and so on) as this fits with their testing schedule. Tests each build against the newest hardware and build release as the hardware becomes available to them. Only updates a system if a specific problem is seen or the the security risk is deemed high enough. This customer actively leverages the API/ABI promise by updating only the parts of the system that need updating. Often only the kernel.

Customer 2 Takes a Proactive, Active approach to their builds but lets individual business units decide when to apply errata, In practice this ends up being a Reactive, Active policy by the business units. Takes the Red Hat Minimal Build and adds only that functionality which is required. Takes which ever point release is in scope during their build cycle (they have taken 5.1, 5.4 5.7 and 6.1) Maintaining only current and previous Major release. Tests each build against all hardware in the estate and against their virtualisation platform. Updates only their builds and allows the Business Units to continue with BAU unless there is an urgent security fix or a specific problem is seen. Again, actively leverages the API/ABI promise by only updating the parts of the system that need focus. In both cases, each customer maintains a test harness of their active builds. The proactive changes are soak tested against the operational requirements using an automated test suite. Reactive changes are dealt with slightly differently at each customer. Customer 1 requires that all changes are tested before a change request to the live system is raised. Customer 2 will often allow an errata to be applied directly to the affected system if we can demonstrate that the observed symptoms are the same and the erratum has no reported regressions. Recently released errata or hotfixes undergo the usual testing. Both of these customers are only able to take this very agile approach because they configure and test kdump as part of their basic build. Errata management does not begin at the point a system fails, it has to be considered in the approach taken to the initial build. These organisations obtain the following benefits by managing their patching strategy effectively: reduced overhead in systems administration clear guidelines on when to pro-actively apply errata swift resolution to unexpected panic events reduced costs to patching with a targeted build automation of patching after testing greater return on investment on existing infrastructure improved uptime to the business units.

If you require in depth assistance with your patching strategy, please consider contacting Red Hat Global Professional Services
(http://www.redhat.com/consulting/)

who are able to help you get the most from your investment in Red Hat Enterprise Linux .

Product(s)

Red Hat Enterprise Linux

Component

patch

Tags

errata

rhel_4

rhel_5

rhel_6

https://access.redhat.com/site/solutions/64094

11/25/2013

What Patching Policy Should I Adopt For My Red Hat Enterprise Linux Servers? - Red H... Page 3 of 3

Comments

Copyright 2013 Red Hat, Inc.

https://access.redhat.com/site/solutions/64094

11/25/2013