Patch management is all about addressing and repairing code and application flaws that can be exploited in a cyberattack. The flaws are inherent to human-written software, so the process of addressing them goes everywhere that software goes. Patch management best practices are about setting a business up for success to effectively repair those flaws across an organization.
Software development is a never-ending attempt to keep products free of flaws, while simultaneously patching those flaws wherever they arise. Keeping software running securely is the key to closing gaps and reducing the organizational risk of an attack. Overall, patch management is a practice that goes beyond just maintaining systems and keeping them productive, but extends to keeping them secure.
This post outlines the basics of patch management best practices, including obstacles to the typical patch management process.
Why do patch management best practices matter?
Besides the paramount aspect of security, patch management best practices matter for a few key reasons.
First, patch management affects system uptime by keeping your software and applications up to date. When software remains unpatched, the consequences of system downtime loom large. By preserving uptime, your business continues moving forward, delivering value to customers and staying active in the marketplace.
Then, if regulatory bodies require that your organization observe certain levels of compliance, patch management plays a role in that compliance. This can apply to government agencies, and in industries such as finance and health care. Mandates often take the form of patching, for example, 90 percent of devices within 30 days of patch issuance, and the remainder within the next 45 days. Many companies observe an internal timeline, holding themselves to an even more stringent standard.
Note the important distinction between patch management and feature improvements; they are (or should be) different work streams in software development. Both features and patches are important, but patches are more urgent than features. It makes sense to include patches and security updates — if they are ready — in a feature release, but it does not make sense to delay security patches until a feature release.
Patch management as part of vulnerability management
Vulnerability management has similar goals to patch management in that it seeks to address holes and flaws where an attacker could exploit something to gain unauthorized access. However, not all vulnerabilities can be remediated, fixed and patched. Some vulnerabilities require more extensive efforts for protection.
For example, a legitimate Windows service could represent a vulnerability. When you install Windows, that service is on and running by default and by design. But then bad actors discover a way to use that service for illegitimate ends, and the service entails unintended consequences. Is there a patch for that? No, because you as IT administrator determine whether you need that service running on that machine; if not, you disable it, but you don’t patch it.
In another example, organizations such as the National Institute of Standards and Technology (NIST) publish lists with recommended settings for considering a device to be secure. In the context of a government or academic environment, their recommendations outline what to enable, what to disable and robust settings for certain registry keys. That is vulnerability management through configuration, but it has nothing to do with patch management.
Or consider a systems management appliance built around an open-source OS like FreeBSD Linux. To reduce the risk of vulnerability, the manufacturer strips the OS down to only the components needed by the appliance. By running without most of the default configuration, the appliance effectively manages vulnerability.
Thus, enterprise patch management plays an important role in vulnerability management, but only in so far as you can fix the vulnerability. The rest of it is knowing how to configure your devices to achieve attack surface reduction.
What do patch management best practices look like?
Smart companies approach patch management methodically because there is no other way to ensure reach to all devices and to keep all devices up to date.
Identify and inventory your systems and network
Whether you have a management system in place or not, the first step is to discover all devices on your network — preferably through automation — and build an inventory. That is the job of systems management.
Scan systems
The next step is to establish what is going on with those devices and what software is installed. The goal is to identify non-compliant, unpatched, vulnerable devices. Systems management appliances usually include an agent or module that lets you scan a machine against known vulnerabilities.
Set priorities for patches
Many IT administrators suffer from patching fatigue as they survey the landscape of devices and the profusion of updates aimed at them. It’s not uncommon to feel overwhelmed and to wonder where to begin, so setting priorities is useful.
Many systems sort by severity, with some software vendors assigning four or five levels of severity, such as critical, high, medium, low and none. Another useful data point is the Common Vulnerabilities and Exposure (CVE) data generated when a vulnerability is reported. With a system of ranking, admins can decide which patches to schedule now and which ones can wait. In most cases, the severity rank is included in the metadata that accompanies the patches.
Conduct phased patch releases
Some IT teams are slow to patch because patching introduces change, and change occasionally brings unintended consequences. Rather than blindly start patching devices, teams dive into the release notes for a closer look. Naturally, they are afraid that the patch will break something specific to their business, and they dig deep to prevent it.
But in a phased approach, they can limit the blast radius of any unintended consequences. For example, they may go with just straight numbers, like a test bed of five percent of their device population. But in a company with thousands of users, that can still result in a hefty support burden if problems arise. Another type of phased approach is to select particular user profiles, workloads and configurations, then roll out the patch to them and gauge the results. That more accurately reflects a cross-section of devices and variety of side effects, if any.
Test patches before releasing them
It’s always preferable to test software — including a patch — before rolling it out en masse to production. Given their resources, IT may apply the patch to a test machine, then study the change logs to see what exactly has been modified and ensure it won’t affect other applications.
Unfortunately, there’s little room for phased approaches in the case of a zero-day vulnerability. That’s when the word is out and bad actors are trying to exploit the vulnerability right now. When patch testing is not an option, some IT teams forgo testing because of the clear and present danger. They accept the calculated risk associated with the patch and move on.
Automate patching
Patch management is a process ripe for automation. In examining patching as a series of processes, IT teams have gravitated toward a traditional, task-driven approach. Many systematically follow the typical process of discovery, provisioning, establishing line-of-sight, developing a patch schedule, setting up tasks, detecting devices then deploying patches.
But in an IT landscape where most users and their devices are remote, and many of those users are not connected via VPN, that approach is limited. In turning to automation, the questions become “How do you connect to all those systems, manage them and reliably perform those tasks? How do you find all devices wherever they may be? How do you reduce reliance on human intervention?”
As depicted below, a new model of automation is evolving.
In a policy-driven approach, detection is an automatic process. Systems management appliances continually perform detection. If the current device needs a patch, the appliance deploys it, verifies proper patching and operation, and updates the inventory.
That differs from having to wait for the next event as in the task-driven approach. When a critical patch is released, the policy running on a particular device determines the need and installs the patch.
Let’s say you have a testing window where you validate newly released patches. In a policy-driven approach, you make sure your main patching policies include filter logic that only allows for patches that are older than your testing window. For a more granular example, if you internally test new patches for 30 days, make sure your patching filters (used in your main company-wide patching policies) only select patches that have a release date older than 30 days.
The result is increased agility and flexibility while adhering to patch management best practices, without the dependence on schedules, specific events and human guidance.
Patch with user experience in mind
Not all patches are designed for transparent installation. They may cause a performance hit during installation, or force a restart of the device. You can avoid that to the extent that you schedule patch management after productive hours, but some users will inevitably be affected.
There is rarely a perfect time for installing a patch, especially in businesses that run 24/7. Patching with the user experience in mind means providing some flexibility in patch management to work with your users. Examples include allowing users to schedule installation or restart for an optimal time, not forcing updates at 10:00 a.m. on Mondays, and not saturating the network with installation traffic.
Document systems before and after patching
Patch management also includes knowing the metrics on your patching efforts, including number of devices reached, percentage of successful/failed installations and devices still to be patched. Those metrics lead to ancillary efforts such a starting up forgotten devices so they can be patched, and directly contacting users who continually hit snooze on updates.
Disaster recovery
Finally, disaster recovery is the safety net of patch management. Having backup strategies in place will allow you to protect your data and restore it should a security update go wrong and require a reset. While typically outside the scope of patching and systems management, a solid data backup and recovery strategy is critical for successful IT operations.
Patch, secure, and manage every endpoint
What can get in the way of successful enterprise patch management?
In certain contexts, company processes can get in the way of, or at least slow down, patch management. Consider the enterprise that has a change advisory board tasked with reviewing every systemic change and approving it before it can be scheduled. The concept of having a regulated, thorough approach to change management is sound, but some scenarios demand agility in IT and the ability to respond quickly. If the company’s process does not allow for special situations, then it can get in the way of patch management best practices.
How can companies improve their patch management maturity?
These patch management best practices pave the way for tactically maintaining the security, system uptime and compliance all organizations crave. More strategically, though, it’s important to establish broad accountability and stewardship of patch management. Like cybersecurity, this can become everyone’s business. It’s shortsighted to assume that the burden of the entire effort falls to IT.
In cross-sectional testing of patches, for instance, do you have advocates and willing collaborators in all departments? When you need testing, is there an informal network of co-workers and regular users around the company on which you can rely? Involving people that way improves your maturity and reinforces the value of those co-workers to the patch management testing process. It’s also a matter of buy-in, all the way down to the ground level. People must be cognizant of the idea that participating in patch management is the ticket to fostering a more secure environment.
Using these patch management best practices allows enterprises to be proactive about how they secure their systems and defend against vulnerabilities.
