patch management best practices

Transcription

PATCH MANAGEMENT
BEST PRACTICES
A “How to” Guide for
Securing the Enterprise
AUTHORS:
Anne Stanton
President, Norwich Group
Susan Bradley
Microsoft Small Business Server MVP
The Fundamentals of Patching – From a Technical and Historical Perspective
Table of Contents
INTRODUCTION ........................................................................................................................................ 3
WHAT IS PATCH MANAGEMENT?....................................................................................................... 4
HISTORY OF PATCHING ............................................................................................................................... 4
DEFINITION OF MICROSOFT PATCHES ......................................................................................................... 4
WHAT IS A PATCH? ..................................................................................................................................... 5
IDENTIFYING THE FLAW .............................................................................................................................. 6
WHY DO WE PATCH?................................................................................................................................... 6
WHAT IS INCLUDED IN A MICROSOFT PATCH? ............................................................................................ 6
HISTORICAL PATCH PROCESS WINDOW ...................................................................................................... 9
FINDING OUT ABOUT PATCHES.................................................................................................................... 9
THE ROAD AHEAD.................................................................................................................................... 11
CHANGE MANAGEMENT AND RISK ANALYSIS ............................................................................ 13
RISK AVOIDANCE – AVOID PATCHES ALL TOGETHER ................................................................................ 13
RISK AVOIDANCE – IF THEY’D ONLY WRITE BETTER CODE ........................................................................ 14
RISK ACCEPTANCE – THE REAL WORLD .................................................................................................... 14
PATCH POLICIES AND CHANGE MANAGEMENT POLICIES ........................................................................... 15
IDENTIFYING YOUR CHANGE MANAGEMENT TEAM ................................................................................... 16
OUR ASSUMPTIONS ................................................................................................................................... 17
SETTING UP A PATCH TESTING SCENARIO ................................................................................................ 17
DISCOVERY OF ASSETS ............................................................................................................................. 18
NECESSARY PATCHES VERSUS ALL PATCHES ............................................................................................ 20
SETTING UP THAT TEST LAB - UTILIZING A VIRTUAL ENVIRONMENT ......................................................... 20
THE BOTTOM LINE .................................................................................................................................... 21
IT’S ALL ABOUT RISK ........................................................................................................................... 23
THE ADMINISTRATOR’S ROLE ................................................................................................................... 23
IDENTIFY KEY FACTORS ............................................................................................................................ 24
CAN YOU PATCH? ..................................................................................................................................... 24
HANDICAPPING THE PATCHES .................................................................................................................. 25
CASE STUDY—BLASTER ARRIVES ............................................................................................................ 26
RESOURCES FOR THREAT ASSESSMENT .................................................................................................... 30
ADDITIONAL TESTING PROCEDURES ......................................................................................................... 31
TESTING PROCEDURES MAY INCLUDE EFFECTIVENESS TESTING ................................................................ 31
STANDARDIZE YOUR OFFICE ..................................................................................................................... 33
TESTING PROCESSES ................................................................................................................................. 33
ARE WE READY TO PATCH YET? ...................................................................................................... 36
PATCH “AT RISK” MACHINES FIRST ........................................................................................................... 36
USE PATCH TOOLS TO CONFIRM COMPLIANCE WITH POLICIES ................................................................... 37
CAPTURE FEEDBACK................................................................................................................................. 38
ENFORCEMENT ......................................................................................................................................... 38
SAMPLE TEST LAB SECURITY POLICY....................................................................................................... 39
TECHNICAL ENFORCEMENT ...................................................................................................................... 42
SET THE POLICY TO MAKE IT EASY TO UPDATE AND COMPLY WITH........................................................... 44
STUFF HAPPENS ........................................................................................................................................ 44
BE WILLING TO GATHER INFORMATION .................................................................................................... 45
CHANGE MANAGEMENT FOLLOW-UP ....................................................................................................... 45
SO I CAN’T PATCH, NOW WHAT DO I DO? .................................................................................................. 46
THE STORY OF TWO PATCHES ................................................................................................................... 47
Ecora Software Corporation
Patch Management Best Practices: A “How To” Guide for Securing the Enterprise
Page
1
THE COMING THREATS .............................................................................................................................. 48
THE BOTTOM LINE FOR YOUR BOSS ........................................................................................................... 49
LIFE BEYOND PATCHING MICROSOFT........................................................................................... 51
PATCHING THIRD PARTY APPLICATIONS ................................................................................................... 52
PATCHING LINUX DISTRIBUTIONS ............................................................................................................. 58
BUILDING PACKAGES FROM SOURCE......................................................................................................... 66
DEPENDENCIES ......................................................................................................................................... 66
SUN SOLARIS ............................................................................................................................................ 67
JUSTIFYING PATCH MANAGEMENT—A BUSINESS TUTORIAL............................................... 69
MANAGEMENT AND PATCH MANAGEMENT .............................................................................................. 69
AN OFFICER’S PERSPECTIVE ..................................................................................................................... 70
INDUSTRY REQUIREMENTS/STANDARDS .................................................................................................. 75
COMPLIANCE WITH ISO900X ................................................................................................................... 81
COMPLIANCE WITH INDUSTRY ASSOCIATION STANDARDS AND REGULATIONS ......................................... 81
RETURN ON INVESTMENT (ROI)............................................................................................................... 82
WHAT DO WE REALLY NEED? ................................................................................................................... 84
DEFINING THE PATCH MANAGEMENT BEST PRACTICES—FIRST ROI VARIABLES PROJECT..................... 85
ONGOING MAINTENANCE AND MODIFICATION—SECOND ROI VARIABLES PROJECT ............................... 85
WHEN A BREACH OCCURS – BUSINESS CONTINUITY—THIRD ROI VARIABLES PROJECT ........................ 86
FINDING TIME IN A CRISIS MANAGEMENT ENVIRONMENT ......................................................................... 87
PURCHASING PATCH MANAGEMENT TOOLS .............................................................................................. 87
VENDOR ISSUES........................................................................................................................................ 88
CONCLUSION ............................................................................................................................................ 89
APPENDIX A– PATCH TESTING .......................................................................................................... 90
APPENDIX B—SAMPLE BUDGET REQUEST ................................................................................... 91
GLOSSARY ................................................................................................................................................ 92
NOTES ........................................................................................................................................................ 93
Page
2
Introduction
ver a late January weekend in 2003, ATMs for well-known financial institutions became
unavailable, flights were canceled due to online ticketing and electronic check-in
problems, and a major city’s emergency 911 network went down. These were just some
of the issues that caused a crisis throughout the world and sent IT staffs scrambling to figure out
what was happening.
O
Pretty soon, we all learned what the culprit was. “Computer worm hits the net,” screamed the
headline.
SQL Slammer, Sapphire, W32.Slammer whatever you want to call it, was tiny as worm files go,
only 376 bytes of code designed for speed. Typical computer transactions involve a “hand shake”
transmission process. One party offers a connection, another party accepts, and the transmission
proceeds by means of traditional TCP/IP processes. SQL slammer, however, used another
transmission standard. It transmitted UDP packets only, through a connectionless transmission.
This worm did not wait for a response. It flooded all vulnerable connections it could find.
That high-speed little file was looking for a port that had behind it, ready and waiting, a listening
application. The Internet Assigned Numbers Authority (IANA.org) maintains the listing of
computer ports used by programs, applications, and typical connections on the Internet. A
computer system has almost 65,000 ports to transmit information back and forth. Typically, these
ports sit there waiting, but sometimes they are in “listening” mode, waiting to be called upon.
Most worms would typically try to find vulnerable systems on well-known ports, those ports from 0
to 1023. SQL slammer was different. It aimed at a port not used in previous attacks, port 1434.
This is a port used by database programs such as Microsoft SQL server and something called
MSDE or Microsoft SQL Database Engine. Microsoft SQL Server is a very powerful database
program typically run on maintained and monitored servers. The other, MSDE, it a small but
powerful database engine used by developers in many applications. Furthermore, port 1434 is
unique. It is not a port used to transmit data; rather it monitors SQL transmissions. All Microsoft
SQL servers listen on this port. Not all MSDE installations do however.
The tiny worm’s tale includes a couple of other twists. Developers use MSDE in many
applications, but do not necessarily tell purchasers that their software uses MSDE to keep track
of something needed for the application’s operation. At the time Slammer struck, patching SQL
server or MSDE was difficult and cumbersome, needing the patch installer to understand SQL
instances. While the original patch to fix the vulnerability came out in July of 2002, the rollup
service pack had just recently come out about three weeks before Slammer appeared.
In early 2003, three weeks was not enough time to have server admins or application developers
test and install a service pack. Database administrators were (and still are) reluctant to install
patches on working databases, so installing the rollup pack was not a priority. Furthermore, all the
unpatched computers equipped with MSDE were primarily in installations where the administrator
had no idea that he or she had MSDE installed. Their software vendors had not informed them,
nor did they have a tool to identify machines that were running MSDE.
Thus, the stage was set for the worm: unpatched machines, unidentified machines that were also
unpatched, a worm built for fast connections, a port never used before for worm attacks, and a
port not used for data, only monitoring. Overall, we had a “perfect storm” for unleashing the worm.
The news reports indicated that the major ISPs and backbone providers of the Internet knew
within minutes that something was up. Realize that unlike Code Red that took 24 hours to go
around the world infecting the globe, SQL slammer was around the world in 30 minutes.1
Page
3
Chapter 1 – What is Patch Management?
W
e define patch management as “the act, manner, or practice of managing, handling,
supervising and controlling the application of code to software in order to fix a bug,
especially as a temporary correction between two releases.”
Many in our industry consider that patch management means applying security fixes that are
typically not temporary corrections. In fact, patch management and its processes include the
entire area of introducing “new code” into an existing environment.
While this document focuses primarily on tools used for testing, supporting, and evaluating
Microsoft security patches, the basic concepts are the same for all software systems. IT
managers and admins must logically control the introduction of new code into a network.
Computers drive world business. No longer can a financially healthy firm exist without technology.
Further, maintaining, protecting, and securing computer systems is, as recent legislative
initiatives make clear, simply good business. A computer, like any mechanical device, requires
regular maintenance. Applying patches to systems that connect to the Internet is simply and
fundamentally mandatory. Period
This document focuses only on the processes and procedures for patch management. We
assume that you have taken steps to harden systems,2 apply firewalls, and install antivirus
protection. Patch management is only one of the steps needed to protect an environment, but it is
an essential element nonetheless. We also assume that you understand the “business” reason
for patching and that you have the proper buy-in from management to add patch management to
your security processes.
Some security vendors recommend installing the bare minimum of patches; however, that still
means that you must rely upon consistent and appropriate processes and procedures.
History of patching
The CERT Guide to System and Network Security Practices3 has small sections on software
patches. The author, Julia Allen recommends applying patches on redundant or duplicate
systems before applying them to main production machines. Any vendor service level agreement
(SLA) should clearly state that firewalls should remain in place during installation of operating
system patches. Allen further argues that those in charge of firewall operating systems and
software need to review those devices as well for updates. Not too long ago, “best practices”
meant not applying software patches in a piecemeal manner, but waiting for the cumulative
security patch that, presumably, was more tested. Now, delay and deferral invite crisis.
Definition of Microsoft patches
In the Microsoft world, patch management included all of the following types of new code
introductions:4
• Critical Update – this is not a security update but a fix for an issue in broadly applied
software. It is publicly available and has an accompanying knowledge base article.
• Driver Update – this updates software that supports and controls hardware. Driver updates
may come from either the software vendor or the hardware vendor.
• Feature pack – software package that includes non-critical additions to the base software
program. It typically appears between major releases.
• Hotfixes – patches built to address specific issues. Recipients may not distribute hotfixes
outside their organizations without written authorization from Microsoft. Get them FREE
by calling Microsoft Product Support Services. Hotfixes do not receive the regular testing
Page
4
process.
• Security Update – this is what we traditionally refer to in patch management circles. There
is a severity rating included with each update, as is a security bulletin discussing the
issue and a Knowledge Base article describing the patch in detail.
• Service Packs – are cumulative packages of hotfixes, security updates, critical updates,
and updates. A service pack undergoes both internal and external testing.
• Software Update – is any update, update rollup, service pack, feature pack, critical update,
security update, or hotfix.
• Update – addresses a non-critical, non-security issue.
• Update rollup – cumulative package of hotfixes, security updates, critical updates, and
updates, collectively tested for easy deployment.
• Upgrade – this software updates and upgrades an application to a newer version while
keeping the settings and data from the prior program.
Each of these categories requires the same process and procedures of testing, acceptance, and
management sign off. Each must go through a process no matter the size of an organization. We
will spend the bulk of our time in this document on security updates.
What is a patch?
Software refers to the instructions mechanical devices receive to process commands in a certain
manner. Typically, developers write software to perform a process in a prescribed fashion.
However, as with any process deriving from human intelligence, there can be incorrect
assumptions made, and the software might not perform as intended.
As Bruce Schneier and Niels Ferguson state in Practical Cryptography –
Most engineers have to contend with problems like storms, heat, and wear and
tear. A bridge designer only has to worry about three threats – water, gravity and
wind. All of these factors affect designs, but their effect is predictable to an
experienced engineer. (This is) not so in security systems. Our opponents are
intelligent, clever, malicious, and devious; they’ll do things nobody had ever
thought of before. They don’t play by the rules, and they are completely
unpredictable. That is a much harder environment to work in.5
A software designer has to worry about threats from an unlimited number of vectors. This means
that the network administrator must consider such risks from these same threat vectors when
analyzing a network’s need for patches. Can the threats come from outside the organization or
only from inside? What potential attack method might exploit a particular flaw? In a later chapter,
we discuss resources for determining these threat vectors.
Software flaws threaten an organization in various ways. The most critical give an attacker full
rights to a system. Many of these flaws stem from buffer overflows. Traditionally stack-based
buffer overflows have been the largest category of security issues, and are places in the software
where more data enters the system than the software is asking for. If the software designer did
not anticipate this, the system would “crash.” Perpetrators target buffer overflows to dump the
processes to ensure that the system remains at an “administrative privilege” level, thereby forcing
the system into “handing over the keys to the kingdom.” When a patch bulletin indicates that the
worse case scenario is that the attacker can “run code of his choice,” this is the equivalent to that
person logging in as administrator to a system.
Software patches themselves are also threats. After each security bulletin release, even with the
rigorous testing done by the vendors on those applications typically broken by previous security
updates, issues still occur. Documentation accompanying a security update addresses any
known issues likely to develop following release of the update. Remember that issues directly
Page
5
caused by security updates qualify for a no-charge support call. In the United States, call 1-866PC-SAFETY.To resolve International issues contact a local Microsoft office.6
Testing patches, ensuring that they do not adversely affect systems and that they protect systems
as intended, as well as applying patches, requires approval from management and may even
require approval from critical line of business vendors. We discuss change management
processes later and in detail. Nevertheless, you need adequate resources if these testing,
ensuring, and applying processes and procedures are to work correctly.
Identifying the flaw
Very briefly, software vulnerability begins when someone looks at code or attempts to reverse
engineer code. While Linux and its variants work under the Open Source licensing model wherein
the source code accompanies the product, Microsoft does not release its source code. Thus,
many security researchers use various techniques to identify flaws. In some cases, the interaction
and connectivity needed between a UNIX or Linux system may provide clues to a researcher for
potential flaws. Other freely available tools include Dave Aitel’s SPIKE (www.immunitysec.com),
Todd Sabin’s DCE-RPC tools (razor.bindview.com), Netcat
(www.atstake.com/research/tools/network_utilities), Ethereal (www.ethereal.com), and many of
the utilities at the Sysinternals web site (www.sysinternals.com).
If the researcher or security company agrees to responsible disclosure techniques, it contacts the
software vendor ahead of time and allows the vendor to correct the flaw or respond to the issue.
Eeye.com7 is one vendor that notifies but does not disclose a flaw publicly at any time prior to the
release of a security update. Irresponsible vendors and security researchers post the vulnerability
to listserves such as Full Disclosure8 along with information that provides a “proof of concept” that
has later been exploited by others and turned into automated exploits. Once contacted, the
vendor reviews the reported information to see if the issue truly is a security flaw. If there is an
issue, the process of building the patch and testing the patch begins.
Why do we patch?
It is obvious that we patch because software is not processing commands correctly. This misprocessing could range from elevation of privilege to information disclosure. Threat Modeling, a
text that explores what an adversary might attain by exploiting a flaw defines the following threat
categories9
• Spoofing identity
• Tampering with data (also called integrity threats)
• Repudiation
• Information disclosure
• Denial of service
• Elevation of privilege
Patch management ensures that correct code replaces incorrect code. However, it is not the only
way to reduce risk. The patch management process also includes mitigation techniques that are
not actual patches but include additional procedures to protect networks if the patch is not
available, or if admins cannot apply it to a network, or if there are other reasons that preclude
applying the patch.
What is included in a Microsoft patch?
Let’s roll up our sleeves, get technical, and examine what is included in each type of Microsoft
patch10. Security patches, critical updates, updates, update rollups, drivers, and feature packs fall
into the general distribution releases (GDR) category. These go through testing across different
platforms and applications to ensure proper functionality, and that the program or update that
includes new features performs as intended. However, Hotfixes developed by Microsoft Product
Page
6
Support Services for a specific situation are not as tested as those included in general distribution
releases. Microsoft Knowledge Base articles, freely available from Microsoft Product Support
Services, always accompany these QFEs.
In the Windows 2003 Server environment and now in the Windows XP sp2 workstation, the
product update packages may include two or more copies of the same files to support two
different types of install environments for a system. When the security patch, critical update,
update, update rollup, driver, or feature pack install, the installer package looks to see what files
already exist on a system. Possible install environments include:
• GDR environment
o Original released version (RTM)
o Service pack version
o General Distribution release
• QFE environment
o Hotfix
Having discovered the appropriate environment, the installer package installs the applicable file
set. To see what version of a file exists in a Windows 2003 server environment, review the
following formats:11
File Version
Source of file
Srv03_rtm.mmmmmmnnnn
This file is from the original RTM version of the product and has not
been updated by any security patch, critical update, update, update
rollup, driver, feature pack or hotfix.
Srv03_gdr.mmmmmmnnnn
This indicates that the file is from a security patch, critical update,
update, update rollup, driver, or feature pack and has not been
updated by a hotfix.
Srv03_spx.mmmmmmnnnn
This indicates that the file is from a SP and has not been updated by
a security patch, critical update, update, update rollup, driver, and/or
feature pack.
Srv03_qfe.mmmmmmnnnn
This indicates that the file is from a hotfix.
In our server, we can see that the file on our server is a GDR version. Thus, it indicates that the
patch engine did not find a hotfix and instead found a GDR version.
For example, let’s look at the file included in Security bulletin 04-024 (04 for the 2004 year, -024
th
meaning the 24 bulletin of the 2004 year). Find this bulletin at
http://www.microsoft.com/technet/security/bulletin/MS04-024.mspx and the sample below is the
patch for the Windows 2003 platform. It includes updates to one file shell32.dll. Inside the installer
package are two files. One expects that the server will still have one of the original dll’s
categorized as a GDR package the other anticipates a hotfix.
13-May-2004 00:07 6.0.3790.168 8,168,960 Shell32.dll RTMGDR
This version is used to apply to servers that have original released version (RTM), Service packs
or General distribution versions.
12-May-2004 23:29 6.0.3790.169 8,168,960 Shell32.dll RTMQFE
This version is used to apply to servers that have received a hotfix version.
Page
7
The shell32.dll File Version Window
While security bulletin 04-024 includes an update to only one file, many patches contain a series
of files that replace existing files on a system. Other security patches may include a series of files
needed to correct the condition. In the Security patch Microsoft Security Bulletin MS04-022:
Vulnerability in Task Scheduler Could Allow Code Execution (841873)12 the patch includes a
series of files needed to remove the vulnerability from the system:
Date
Time
Version
Size
File name
Folder
08-Jun-2004
08-Jun-2004
03-Jun-2004
08-Jun-2004
08-Jun-2004
08-Jun-2004
08-Jun-2004
08-Jun-2004
08-Jun-2004
18-May-2004
22:01
22:01
22:54
22:01
22:01
22:02
19:59
22:02
22:02
03:46
5.1.2600.105
5.1.2600.155
5.1.2600.155
5.1.2600.122
5.1.2600.155
5.1.2600.1564
5.1.2600.1564
5.1.2600.1562
5.1.2600.1564
5.1.2600.1555
48,640
251,392
9,728
301,568
159,232
260,096
10,752
306,688
172,544
593,408
Browser.dll
Mstask.dll
Mstinit.exe
Netapi32.dll
Schedsvc.dll
Mstask.dll
Mstinit.exe
Netapi32.dll
Schedsvc.dll
Xpsp2res.dll
RTMQFE
RTMQFE
RTMQFE
RTMQFE
RTMQFE
SP1QFE
SP1QFE
SP1QFE
SP1QFE
SP1QFE
Applying new executables and dll files introduces change into a stable system. As evident from
the files listed above, the security update includes both executables and dynamic link library files.
An exe file is a file that a computer can directly “run” or execute. A DLL file contains a range of
functions accessed by other Windows applications. The standard functions in the Windows
Application Programming Interface (or API) are accessed using DLL files. This standardization
eases collaboration among disparate applications. Without these building blocks, applications
Page
8
would look and act much differently. A DLL can have the extension of .exe, .dll, .drv, or .fon. In
any case, patching introduces new files and new code into a stable system. Thus, test to ensure
that you have tested the install and uninstall processes, as well as any potential rollback issues.
Historical Patch Process window
Until recently, an administrator could be somewhat lax in applying patches. According to
Forrester Research, the average time between the release of a patch and the attack of a worm
was 305 days in March 2003. However, that window of opportunity has been shortening.
Common
Name
SQL Slammer
Bugbear
Frethem
Yaha
Elkern
Klez
Badtrans
Historical window of patching in 200313
Attack
Patch
Advance
Impact of Attack
Date
Issued
Notice
Infections doubled every 8.5
1/25/03
7/24/02
185 days
seconds
More than 2 million infected
9/30/02
5/16/01
502 days
computers
12 variants in the first two months
7/17/02
5/16/01
427 days
of activity
Intercepted in one of every 268
6/22/02
5/16/01
402 days
emails at peak
Detected in more than 40 different
4/17/2002
5/16/01
336 days
countries
$9 billion worldwide productivity
4/17/02
5/16/01
336 days
loss
Message Labs has seen 458,359
11/24/01
5/16/01
192 days
instances
Nimda
9/18/01
10/17/00
336 days
Spread worldwide in 30 minutes
Code Red
7/19/01
6/18/01
31 days
Infection doubled every 37
minutes
As the table shows, in March 2003, administrators had many opportunities to test patches and
even wait until a Service Pack before deploying. However, this window has been shortening in to
include instances where patches were not available. More recently, the time between the patch
and the worm for an exploit commonly known as MSBlaster was 16 days. In June 2004,
Microsoft’s Internet Explorer browser suffered several security issues left unpatched by Microsoft
for many weeks. Therefore, while stressing patch application as the best security prevention, we
include remediation techniques as well in this patch management process.
Finding out about patches
We next need to identify resources that help identify which patches an organization needs.
Whether you patch manually or use patch management tools, ensure that your team or your
Security officer is aware of the patches available for the products in your network. For most
operating systems, this is rather easy to do, as all major vendors have e-mail or RSS notification.
For Microsoft products, subscribe to get e-mail notifications at
www.microsoft.com/security/bulletins/alerts.mspx.
Page
9
Sample of a Security bulletin
Another way to stay current is to sign up for RSS or “really simple syndication.” Using a feed
reader, you can receive these security bulletins quickly and easily. For more information about
choosing and installing an RSS reader, go to
www.microsoft.com/technet/security/bulletin/secrssinfo.mspx.
Page
10
RSS feed of Microsoft Security bulletins (Outlook 2003 using Newsgator)
Microsoft moved to a once a month patch schedule that has patches releases on the second
Tuesday of the month between 10:00 a.m. and 11:00 a.m. However, there may be times that a
patch will be released “out of band” if an active exploit is “in the wild.”
For the Red Hat Enterprise Linux platform, there is also many ways to obtain notifications directly
from the vendor by subscribing to email or an RSS feed at the following web site:
www.redhat.com/security/team/advisories.html14
RSS feed of Red Hat security bulletins (Outlook 2003 using Newsgator)
Software vendors often provide additional support resources. For some applications installed on
your network, it may be more difficult to track down notification sources. Secunia.com, for
example does provide security information for quite a few vendors at secunia.com/vendor/. For
your “line of business” applications, you will need to contact the vendors regarding the manner in
which they notify customers.
The Road Ahead
In the coming sections we will discuss patch methodologies, risk management as applied to
patching, and development of a patch team. We will analyze a security bulletin to find resources
to determine if exploit code is already on the Internet and thus having an impact on a patch timing
decision. We will discuss ways to set up a lab environment for testing patches and the
procedures that you should review when ensuring that the security updates perform as expected.
We will point to resources for proof of concept code or other techniques you can use to confirm
patch status. We will discuss ways of enforcing patch management policy. We will discuss cases
where you cannot patch because of vendor limitations or the lack of a patch, and what mitigation
information and resources you can look for. While we focus primarily on Microsoft technologies,
we will be vendor neutral in the use and recommendation of patching tools and give you
Page
11
resources to assist you in making your choice.
Whatever technology or process you use to manage patches, both the application and the
reporting are key elements in ensuring that systems remain safe and secure. Ensuring that the
patch or mediation technique performs as expected is central to any patch strategy.
Page
12
Chapter 2 – Change Management and Risk Analysis
T
raditional risk analysis accepts risk as possible, calculates the associated costs, and either
applies mitigation techniques and procedures to reduce exposure, or transfers the risks to a
third party. This perspective bears directly on how firms approach the patch management
process.
To give this perspective meaning, consider that a century or more ago, when foot travel
dominated, diseases replicated slowly. Historically, epidemics took time to replicate throughout
the population. Today, given airline travel and international trade, we now accept that diseases
travel the globe in a relatively short time. However, imagine a person contracting a highly
infectious disease for which there was no cure, and that this person could hop on a plane, travel
the globe, and infect people throughout the world in less than 30 minutes.
This is exactly what SQL Slammer did. It was a very tiny bit of code, one “sneeze” or a “cough” (if
you will) that had several things going for it. First, while the patch [or inoculation to keep with our
disease analogy] was available many months before Slammer launched, the “cumulative
inoculation” or Service Patch came out only about 3 weeks before, not enough time (back in
2003) for system administrators to get the patch through change management procedures.
As a rule, database administrators are probably reluctant to implement change management
because they are suspicious of potential impact on their systems and because, until recently,
SQL Server patching was extremely cumbersome to accomplish. Add to this the fact that patches
affect both the full SQL server database system AND a “lite” version of the product embedded,
without fanfare, in several other products, and it becomes easier to understand why some
database admins might appear to be Nervous Nellies.
This reluctance exposed asset management systems to a risk that Slammer readily exploited.
Essentially, Slammer was a bit of code built to transmit over UDP protocol, a “connectionless”
worm that did not wait for the syn-ack to occur and merely transmitted packets without waiting for
confirmation that someone or something was on the other end of those IP addresses. Then it
required that port 1434 be listening. Up to that time, this had not been a targeted port. Previous
large-scale attacks always probed on port 80, the one that web sites use.
Even with its small bit of code—built for speed—Slammer had a tremendous impact. Consider the
pedigree of businesses whose lack of patch management was the subject of public commentary
worldwide. Continental Airlines had to revert to issuing paper tickets; Bank of America had to
rescue its ATM machines; and even Microsoft coped with security breaches.15 While one could
argue that a lack of network segmentation contributed to the chaos (we discuss network
segmentation later), the fact is that Slammer disrupted a trusted system. “Business as usual” was
no longer business as usual for many companies.
Whether those companies realized it or not, they accepted the risk associated with connecting to
the Internet, and accepted the risk of not patching—waiting, instead, for the Service Pack
approval process to work its way to completion. Currently, we have less than two to three weeks,
maximum, to approve the latest patches before an exploit arrives. In the best firms, this window of
risk management evaluation is less than a week.
Risk avoidance – avoid patches all together
Is there any way to avoid patching altogether? If your firm does not connect to the Internet and
allows no introduction of outside destructive software, your firm could actually install a base
image of an operating system and apply only patches needed for functionality improvements and
never worry about security updates. However, can you guarantee, without ANY doubt that no
software will bridge the “gap of air” between the outside infected world and the inside network?
Page
13
Again, think in terms of our heath and epidemic analogy. If you have a child of school age, do you
refuse to have preventive medical treatment in the forms of shots, convincing yourself that your
child will never have contact with any other infected person, or do you understand that prevention
(also known as shots) are a part of life and growing up and must be done. Most security
benchmarks, including those from National Security Agency, Control Security Services16and the
Center for Internet Security,17 recommend keeping computers and devices up to date with
security patches. If you wish to use alternatives to patching, making sure that your remedial
actions are effective should be part of your standard testing process.
Risk avoidance – if they’d only write better code
As noted in a previous chapter, the fact that software designers have to imagine threats from all
sorts of vectors means that no program that interacts with humans can be completely free of
issues. As we ask applications to produce more real time information and to be predictive, and as
we add everything from information retrieval to real time analytics to RFID tag identification, we
ask more and more of our systems and add more complexity, not less. If your attitude is that
patch management issues are a temporary thing and once we get a “proper” operating system we
won’t have to patch anymore, then, in my opinion, you have the wrong attitude.
Risk acceptance – the real world
In May 2002, CIS quoted the Gartner Group study asserting, “Over 90% of cyber attacks exploit
known security flaws for which a patch is available”.18 Patching and other system hardening
measures effectively mitigate 83% of security issues overall. Accepting the probability of risk and
then installing all necessary patches ensures that you limit your exposure to vulnerabilities
without relying on blocking mechanisms. It is also important to note that confirming patch
installation is a critical part of this procedure.
Here again is our risk calculation model, expressed as ARO x SLE = ALE with the components
defined as follows19:
Abbreviation
Defined as
Example
ARO
Annualized Rate of
Occurrence
3
Multiplied by
SLE
Single Loss Expectancy
$26,000
Equals
ALE
Annual Loss Expectancy
$78,000
So we’ve done our risk calculation shown above and determined that any costs incurred less than
our calculated Annual Loss Expectancy of $78,000 will be a potential cost savings to our firm.
Choosing to accept the probability of risk, we choose to apply security patches.
Within the U.S. health care system, the Food and Drug Administration must approve all drugs
before their distribution to the public. Similar to testing and approving drugs, we must formalize
change management processes in our own organizations.
Every firm needs a person or a team that signs off on and authorizes change. President
Truman’s desk had a plaque that read, “The buck stops here.” The same is true for patch
management. The person or team must fully understand the firm’s IT environment. There must
be written policies, procedures, and sign-off checklists. We discuss policies and procedures
BEFORE discussing the patch analysis process for a very good reason: You must have the policy
in place before implementing the technology.
Page
14
Patch policies and change management policies
Patch policies range from informal guidelines to formal documents identifying authorized patches
that appear to function as expected, and that do, indeed, protect the vulnerability they are
supposed to protect. Formal procedures may also call for processes that verify and confirm patch
installation. Many medium and large organizations require a procedural type of policy that guides
the patch management team through the patch approval process. Samples of patch policies or
change management policies are available at the following locations:
•
•
•
•
•
•
•
Sample of change policies at the Stanford Synchrotron Radiation Laboratory
http://smb.slac.stanford.edu/blu-ice/dcsAdmin/node6.html
Sample of change policies at Stanford’s University ITSS department
http://www.stanford.edu/services/changemgt/itss_change_policy.html
State of Connecticut Management Advisory committee Change Management policies
http://www.cmac.state.ct.us/policies/change.htm
Texas A&M University Change Management Policy
http://www.cis.tamuk.edu/help/policies/1_050_Change%20Management%20Policy.pdf
IT Assessments, tools for Project Management http://www.ittoolkit.com/cgibin/itmember/itmember.cgi/reg/ITBE-ok
University of Texas Health Science Change Management Policy
http://www.uth.tmc.edu/itsecurity/policy/Change_Mgmt_Policy.htm
Use a search engine to search on the words “Change Management Policy” for additional
resources
In general, these policies include the following elements:
•
•
•
•
•
•
•
•
•
•
•
•
Identification of the assets that need patching
Identification/preparation of images that properly represent either the exact systems in
your firm or a close approximation
Testing of impact on applications
Patch installation on the test systems
Patch verification of install
Optional verification of effectiveness
Issue management
Assessment of impact
Remedial actions if impact does not affect all systems
Follow-up with vendors
Sign off of the patch
Determination of timing of change implementation
Notice that we have performed all of these procedures and have not even begun to roll out these
patches into our production network. Further, after you have cleared the “patch” for installation
across your network, you need to identify the best deployment mechanism. Additional discussion
of change management techniques is available at
http://www.microsoft.com/technet/itsolutions/techguide/msm/smf/smfchgmg.mspx. More details
about these steps are forthcoming in the next sections of this document.
One resource and guidance for patch procedures and timing of patching is the SANS.org @Risk
newsletter that includes information regarding security vulnerabilities and how “Council sites”
react to them. Find subscription information at the SANS.ORG web site:
http://www.sans.org/newsletters/. As a preview, consider the following information contained in a
recent newsletter issue:
Page
15
Sample of the @Risk newsletter from the SANS.org organization
Ensuring that your organization has policies and procedures to introduce “change” into your
organization in a controlled and managed fashion is important in managing your risk of patching
as well as mitigating dangers exposed by vulnerabilities.
Identifying your change management team
When setting up the team that will approve and administer your patch management process, stay
sensitive to the needs of different divisions, departments, applications that are key to your
organization. Similar to the process that every organization should have done in advance of the
Year 2000, revisit those procedures and identify those key technology assets that you rely upon.
Ensure your change management team includes a representative of each critical technology. If
you rely on Oracle or SQL Server as the underlying backbone of your firm’s databases, ensure
that you have a technical representative on your team. For these key elements of your firm’s
Page
16
infrastructure, you should also ensure that your team has vendor contact information to follow up
on issues that may arise during the patching process.
Our assumptions
We assume that concurrent with your resolve to establish a patch management process, you
have established methodology and plans for true disaster recovery. You need a tested backup
procedure that gets you back to the exact copy of any system before you began the patching
process. While traditional tape and file-by-file backups give you roadmaps to where you were,
most computer consultants and IT professionals would recommend that you look into Drive
Imaging, whereby an exact copy of the data you have is copied in a snap shot format such that
you can reapply that image and be back in “business” reasonably quickly. Even if you test the
patches on test boxes and they appear to work flawlessly, never apply patches to a key critical
business computer asset without ensuring that you have a backup of those files mandatory to
restoring that system to full operational status.
We also assume that you do the minimum amount of testing of that backup and recovery process
to assure yourself that the restore process of even a few files is successful. Rename a few files of
varying sizes and restore those files using your restoration software program. If you have a drive
image, test the restoration of these devices by reapplying the image. Just because the backup
has run and the log file indicates that the backup was successful does not mean that a restoration
will be successful.
Setting up a Patch Testing Scenario
By this time, you and your managers need to have agreed to introduce “managed change” into
your organization. A first and fundamental step in this process is setting up a test lab. You will use
this environment for initial testing of all patches. Do not test patches on machines that contain
real or original data (normally called “production systems”). As we will illustrate throughout, a
reasonable test lab is well within the reach of nearly all budgets. This test environment should
mimic, as much as possible, the setup and structure of your production network. Readily available
tools and procedures help you set up virtual machines that mimic your real organization.
Your organization might allow for additional testing on machines run by a team of users in an
actual production environment. In this situation, identify certain staff positions that use your key
line of business applications to test security patches and changes in live production settings
before pushing patches out to the rest of the network.
To prepare a test lab to reproduce the computing assets you must patch, you need to know what
assets you are protecting. This should be easy—right? All you have to do is count up all those
physical computers and network devices that appear on your asset listing or depreciation
schedule. Either of these shows exactly when and for how much you bought equipment that you
have in your building—right? That written schedule is all you need to identify the hardware and
software your firm has purchased—right?
Not so fast! Multiple recent advances work against your keeping a tight rein on your computer
systems. The first concerns the relative ease of bringing in a piece of equipment from Best Buy or
CompUSA and attaching to the network. Thus, you may not have a Linksys wireless router that
has a vulnerability that needs patching on your asset schedule, but you might very well have one
inside your office. Next is the relative ease of cannibalizing parts from older systems and using
them in newer systems. Your asset schedule may state that you have a 1992 Compaq but, in
reality, while the chassis is long gone, parts of that system may still be in machines in your office.
When we review assets schedules of companies for valuation purposes, we regularly discount as
worthless computer equipment more than four years old. That does not mean the equipment is
not still in your network introducing possible vulnerabilities into your system. A nearly opposite
Page
17
recent phenomenon concerns virtual personal computers and servers that are inside other
machines. If you see only two computers, but your physical scan of the system indicates there
are four IP addresses in use, there may be operating systems you are not aware of inside of
tracked and documented machines.
It is also important that you not assume you know all the software and even operating systems
that are on your networked machines. We have seen cases where IT staffs have introduced
unofficial operating systems into a network without management’s approval. Lastly, any device
that accepts a RJ45 network connection is a device that could introduce vulnerability into a
network. Any device that uses TCP/IP connectivity can be an entry point to a network. Don’t
assume that because a device does not have a monitor and a keyboard that it is safe from
exploitation. BugBear is one example of a virus whose “footprint” was TCP/IP attached printers
suddenly printing out excess paper and garbled characters. Another example is hijacking HP
Webjet or other printer administrative programs for use in attacks.
We assume, of course, that your organization already has a written policy that forbids any and all
of these activities. Samples of these kinds of polices are available at the SANS.org policy project
center at http://www.sans.org/resources/policies/. In general, such policies stipulate that
management approval must precede introducing new assets into a network.
One of the ways to preclude the introduction of rogue software is to standardize on an installation
image of a computer system. Subsequent new assets conform precisely to what the firm expects
and are thus controllable and patchable. A sample of a policy that requires that all introductions of
assets be done in a controlled manner is available at the SANS.org policy project center at
http://www.sans.org/resources/policies/Aquisition_Assessment_Policy.pdf. Not controlling the
introduction of assets into a network greatly increases the potential for unknown vulnerabilities
entering the system.
Discovery of Assets
Your first step in designing an effective patch management program involves understanding the
environment you have to patch. In any sized firm, understanding the operating systems and all
the devices susceptible to patching is pre-requisite to determining which patch tool to choose. For
example, if your environment contains a mixture of Windows and Linux/Unix distributions, choose
a patch tool that supports both platforms. If your environment consists mostly of remote assets,
choose a tool that supports background patching on a variety of connection speeds. You don’t
want the boss dialing in from home on a 56K connection as you push down a large device
update.
Depending on your environment, you can choose among various tools to examine assets. In the
Microsoft environment, choices include tools ranging from Active Directory to identify all
connected assets, to WSH for script a system inventory. There are also many freely available
options, and shared scripts can remotely inventory a network and build an appropriate asset
database. Consider also commercial inventory and control products such as Microsoft Operations
Manager [MOM], Argent, and Tally Systems, just to name a few such tools that have inventory
features.
Sample asset analysis tools include:
• Microsoft Corporation, Microsoft Software Inventory Analyzer
http://www.microsoft.com/resources/sam/msia.mspx
• Lynch, Chris, Inventory every computer in Active Directory,
http://cwashington.netreach.net/depo/view.asp?Index=808&ScriptType=vbscript
• Lynch, Chris, Create Hardware Inventory in Excel Format
Page
18
Host Name:
_____
OS Name:
Microsoft Windows XP Professional
OS Version:
5.1.2600 Service Pack 2 Build 2600
OS Manufacturer:
Microsoft Corporation
OS Configuration: Member Workstation
OS Build Type:
Uniprocessor Free
Registered Owner: ________
Registered Organization:
Product ID:
______________
Original Install Date: 9/27/2003, 8:25:41 PM
System Uptime:
2 Days, 1 Hour, 21 Minutes, 26 Seconds
System Manufacturer: Dell Computer Corporation
System Model:
XPS-Z
System type:
X86-based PC
Processor(s):
1 Processor(s) Installed.
[01]: x86 Family 6 Model 8 Stepping 10 Genuine Intel ~996 MHz
BIOS Version:
DELL - 20010221
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device:
\Device\HarddiskVolume1
System Locale:
en-us; English (United States)
Input Locale:
en-us; English (United States)
Time Zone:
(GMT-08:00) Pacific Time (US & Canada); Tijuana
Total Physical Memory: 511 MB
Available Physical Memory: 114 MB
Virtual Memory: Max Size: 2,048 MB
Virtual Memory: Available: 1,961 MB
Virtual Memory: In Use: 87 MB
Page File Location(s): C:\pagefile.sys
Domain:
___________
Logon Server:
\\__________
Hotfix(s):
8 Hotfix(s) Installed.
[01]: Q147222
[02]: Q823718
[03]: Q832483
[04]: Q828026 - Update
[05]: wm817787
[06]: wm819639
[07]: wm828026
[08]: KB811113 - Service Pack
Network Card(s):
3 NIC(s) Installed.
[01]: 3Com Ether Link XL 10/100 PCI TX NIC (3C905B-TX)
Connection Name: Local Area Connection
DHCP Enabled: Yes
DHCP Server: __________
IP address (es)
[01]: __________
[02]: VMware Virtual Ethernet Adapter for VMnet1
Connection Name: VMware Network Adapter VMnet1
[03]: VMware Virtual Ethernet Adapter for VMnet8
Connection Name: VMware Network Adapter VMnet8
Sample of the information that can be obtained from a system using scripting
Page
19
Your patch management tool may also aid in the asset identification process. If you have SNMP20
enabled on your system, use that to query the devices attached to your network. Remember that
print servers and other devices can be just as vulnerable as computer systems.
Necessary patches versus all patches
When analyzing what to patch, determine your firm’s acceptance of patching. Do you own all of
the assets you need to patch or are you bound by lease agreements to patch only on a schedule
that a vendor approves? Some medical equipment vendors certify their equipment only through
certain service pack levels and thus you may void your agreement if you install patches or even
antivirus products on their equipment. Check your service agreements and vendors for any
caveats they may have that limit your patching ability. You may need to install traffic limitations
and other barrier technologies to ensure that TCP/IP traffic is only in the direction you want it to
be if you are unable to install patches on leased equipment that interacts with owned equipment.
You can also determine the role of the computer or server and determine the patches minimally
needed for that role. Be very careful in using this minimal patch methodology unless you truly
have a clear handle on the software in your network and the traffic that routes to domains, vlans
and other routed networks. Again, using the example of SQL slammer, many desktop users
simply did not know that they had a version of SQL called Microsoft SQL Desktop Engine that
was just as vulnerable to SQL slammer as Microsoft SQL server if left unpatched. Thus, manual
patch methods whereby you push only those patches that you think you need may not be a wise
move at all.
In some cases you might decide that the web server you are patching that does not have
NetBIOS ports open to the Internet does not need the latest patch for NetBIOS information
disclosure vulnerability. Nevertheless, this risk-based patching methodology needs to be
discussed and approved in your organization. You and your team need to weigh and determine
the exposure and risks appropriate to either patching or not patching. There are costs to both
methodologies. We determine the threat vector and vulnerability risk as factors in the timing of
patching, not whether or not the patch will be applied.
Setting up that test lab - utilizing a virtual environment
After inventorying the network, set up a lab that provides the best equivalent emulation of the
actual network. Either by having duplicate machines and hardware, or by using virtual machines
based on VMware21 or Microsoft Virtual PC or Virtual Server, you can test using equivalent data
or even exact images of data in your network.
Dr. Jesper M. Johansson offers the following discussion for using virtual PCs to obtain drive
images of data used in patch testing. (More details on this method and more on network threat
modeling will be included in the forthcoming book by Jesper Johansson and Steve Riley entitled
Protecting Your Windows Network. It will be available from Addison Wesley, Inc. in 2005.)
“There is a neat new way to get a near-perfect replica of a production system that
you can use for testing. Microsoft Virtual PC 2004 includes the ability to make a
new virtual disk based on an existing hard disk. Using this functionality you can
install Virtual PC on a production system (mind you, probably not one that is
currently serving clients) and make a replica of its disk. Then you can copy that
disk image off to another computer. Now install Virtual PC on that other computer
and create a new virtual machine pointing to the disk image you just created. On
the first boot, you will need to remove Virtual PC from the virtual machine, and
possibly install a few drivers. However, you can now set up the disk in undoable
mode and have a copy of a production system on which you can test new
security updates. Obviously, this does not work for all types of testing, such as
testing with particular hardware [that] is not emulated by Virtual PC, but it works
pretty well for testing interoperability with software. Cost-wise it is far more
Page
20
efficient than purchasing and managing additional hardware since Virtual PC only
costs about $150.”22
If you use virtual machines and actual data, be aware of potential security issues of using copies
of data. Ensure that your organization has previously gone through the process of data
stratification. Categorize data as being public, restricted, or confidential.
Do not use servers or workstations that contain restricted or confidential data for testing purposes
unless the change management team stipulates that testing must include this more sensitive
data. When imaging such data, follow strict procedures regarding the handling of such data as
detailed in the firm’s change management documentation. Destroy the imaged data after the
testing procedures and have these procedures monitored and audited to ensure that data
integrity occurs within the organization. You may need to include such data as credit card
numbers in databases in a test patch process to ensure the patch process is successful.
However, most security patches for database engines never touch the data itself. The patches,
typically, restrict themselves to adjusting the code in the database engine only. Testing
procedures that include database integrity (hash totals, for example) may be included in the
change management procedures to ensure data integrity before and after the patching process.
Patching databases is one of the most critical patch implementations in any environment as it
typically involves introducing software into a critical key business asset—the database
information. Thus, meet with your firm’s database software engineers and vendors and get their
recommendations for patch installations and database security. In the history of worms that
affected databases, an SQL database administrator could have thwarted SQL slammer if he or
she had merely taken the time to remove a blank SA password.23 Here is another example of
where adhering to traditional security measures could delay the timing of a patch application.
Again, we recommend against not patching and relying on mitigation alone. We do grant that
following some basic security guidelines will buy you time for patching. Review the resources
listed here to become a smarter patch manager for databases:
SQL Server
• National Security Agency, SQL Server security recommendations at
http://www.nsa.gov/snac/db/mssql_2k.pdf
• SQLSecurity.com at http://www.sqlsecurity.com/DesktopDefault.aspx
• “Best Practices Analyzer Tool for Microsoft SQL Server 2000 1.0”
http://www.microsoft.com/downloads/details.aspx?FamilyId=B352EB1F-D3CA-44EE893E-9E07339C1F22&displaylang=en
Oracle database
• Center for Internet Security, Oracle database security benchmarks
http://www.cisecurity.org/bench_oracle.html
• Oracle Corporation, Security notifications
http://www.oracle.com/technology/deploy/security/alerts.htm
The bottom line
If you are running around once a month feeling as if you don’t have a handle on what you need to
patch, you need to stop and take action now. Based on this first section discussing the technical
aspects of patch management your action items are as follows:
•
•
•
•
Identify key business computer assets
Review the basic security procedures of your organization
Ensure written policies are in place
Set up a patch management team based on that analysis
Page
21
•
•
•
Test or set up backup and disaster recovery techniques
Set a patch management policy based on risk management
Identify the testing lab you will use to perform patch testing
Our next section discusses ways to analyze and review security bulletins. This will help you
identify patch needs based on a risk management determination and develop patching
methodologies. We will also review additional techniques of enforcing patch policy to affected
systems, and preview expected changes in patch management—including a look at the
increasing number of tools that patch both Windows and alternative operating systems.
Page
22
Chapter 3 – It’s all about risk
E
arly evidence of a truly significant patch problem surfaced in mid-2003 postings to public
newsgroups. Windows NT 4 server systems were dying in the water. While they were
booting up and clients were able to log in, none of them could connect to the internet and
the RRAS service was gone. Reports began filtering back to the newsgroups that there was a
problem with a recent security patch. Posters indicated that the support representatives
recommended pulling the patch off and rebooting systems. Removing the one patch confirmed
restoration of RRAS and Internet services. While the support call to the vendor was a free call,
placing it cost down time and investigation time for the administrator or consultant. Three weeks
later, the vendor re-released the patch and all normal server functions performed as expected.
The above accurately depicts what occurred to the Small Business Server 4.5 platform. One
patch caused a service not to run. Microsoft Security Bulletin 03-029’s purpose was to protect
Windows NT 4 from a denial of service attack. Instead, the patch caused a denial of business.24
The vendor identified the problem, repaired the patch, and removed the issue. Why did this
occur? Here is some informed speculation: resources for testing on the NT platform concentrate
on traditional servers and not on the integrated platform of the Small Business Server. For
whatever reason, testing did not find the problem with 03-029. As a result, instead of protecting
the system, the patch actually introduced failures to the system. There was less risk to a network
from a potential exploit than there was benefit derived from the original patch. In this case,
introducing change did more harm than good.
Since that time, Microsoft has increased both internal and external patch testing to reduce these
issues. A recent report card gives a grade of “C” to the current situation of Security patches,25 but
commended Microsoft for developing a process for external testing to better identify patches that
cause issues.
Regardless of changes to vendors’ internal testing methodologies, it remains mandatory that you
test patches before rolling them out to your network, your workstations, or your servers. As soon
as a patch becomes available, the clock begins ticking. Miscreants begin by analyzing the patch,
looking for a potential exploit, and then developing a proof of concept of the vulnerability.
Therefore, the faster you obtain approval for change, the faster you can patch. You need to
control patches and patch installation in your network. Only by testing and anticipating issues in
advance will you be able to manage change in your system.
The administrator’s role
To use a sports metaphor, a patch administrator’s job is to coach on the sidelines, introducing a
new player to the field. Introduce an untested player and the play misfires. Introduce a proven
performer who is ready to go and ready to do his job and the team functions smoothly. To further
this metaphor, it is the admin’s job to understand which player is better in front, which one needs
to be a receiver, which is better at offense than defense. You may want to patch front line players
faster than those who are inside the perimeter. You may want to take mitigation actions on
players that, for one reason or another, are inappropriate for patching.
As coach of your network, you need to know all of your team members well. Using scripts, tools,
and inventory management processes, you should know exactly what computers and programs
you have running. Your first job is to determine what you have and what you need to patch.
Looking at a Microsoft world, you first need to determine workstations and programs. To do this,
use a variety of tools or maybe just script an inventory solution. Available script sources include:
•
•
•
http://www.microsoft.com/resources/sam/msia.mspx
ftp://ftp.microsoft.com/PSS/Tools/Networking_Support/MPSReports
Page
23
•
http://www.belarc.com/ctadvisor.html?B
Available programs include:
•
•
•
Microsoft Systems Management Server http://www.microsoft.com/smserver/
Remedy asset management http://www.remedy.com
BindView control solution suite http://www.bindview.com
If you have a mixed environment, you need to investigate other tools and programs to remotely
manage and inventory your system. You must know what you have in order to control your
systems. Regularly scan your network to detect any new devices present on the system.
The point here is critical: If you don’t have control of your network, you don’t have control of your
network.
Identify key factors
Recall the hysteria before the year 2000 when the world’s computers were going to come to a
crashing halt. Remember also the processes you used to identify key applications that you had to
keep functioning for your firm to survive. It is now time to resurrect those evaluations and
examine all processes and procedures key to your organization. Determine which applications
are so vital and key to your organization that impairing their operations would severely affect your
company.
Can you patch?
So now you have identified all assets that need patching—desktops, laptops, routers, web
servers, firewalls, domain controllers, database servers, SQL servers, line of business
applications--every device in your network that contains computer code needs monitoring and
patching. However, here’s a catch-22 many of us face: we do not truly own the assets we are in
charge of patching. In the health care industry, for example, vendors that own and lease the
equipment point to Federal Drug Administration rules and will not support the installation of
patches. Conversely, the FDA argues that vendors can support patching of systems.
Furthermore, many of us outsource our Information Technology departments and have Service
Level Agreements in place. Introducing unauthorized software via patching, antivirus, or
monitoring software may void agreements.
Equally frustrating is the stubbornness we see among line of business vendors who will not go on
record and support a patch or, worse yet, will not support a service pack or patch on systems
running their software. When Microsoft released Windows XP SP2, many line of business
applications refused to support the installation of the patch.
As coach, then, you find yourself in a truly difficult situation: skip the patch and expose unpatched
vulnerabilities to exploits, or patch and risk voiding service agreements. This is not exactly a great
choice. Some time back a security researcher left the field and, in his parting post to a
newsgroup, he urged those of us who remain to “Demand better security from vendors and hold
them responsible”26 We need to recognize that line of business software vendors impact our
security decisions. Understand this message very carefully: Use what little influence you have
during contract negotiations to make your line of business vendors understand how they affect
your security decisions.
There is no easy answer for those of us in this position. If contractual arrangements constrain
your patching responsibilities, the next section on timing of patching is irrelevant. You will instead
need to concentrate on procedures to reduce exposure in your network. These procedures may
include vlans [virtual local area networks], air gaps, devices that restrict traffic to flowing one-way
Page
24
across your network. Bottom line: if you cannot patch, your job is to isolate, segment, defend, and
minimize exposure as best as you can.
Handicapping the Patches
Once a vendor—Microsoft or Sun or Adobe, for example—releases a security bulletin or patch,
the clock starts ticking. You have a limited about of time to review the bulletin, decide what to
patch, what to mitigate, where your threats are, whether or not there are exploits in the wild and
how they might enter your system. I call this process “handicapping the patches”. In my firm, I use
this process to determine when I will patch. I control my machines and sometimes choose to
install tested service packs before my vendors approve them. I don’t void any warranties, nor do I
jeopardize any contractual arrangements. I could choose not to patch my systems if I felt that I
had enough blocking mechanisms in place. I could use the methodology to identify mitigation.
My small size, my fully enabled active directory environment, and my 100 percent “borg network”
(all of my desktops, laptops, and even telecommuting computers are on the Windows XP
platform) mean that I can quickly and easily identify, patch, and protect my assets. That does not
mean I have any less highly sensitive and critical data. However, I have two issues that expose
me to significant risk.
First, my industry routinely handles information—names and social security numbers—targeted
by identity thieves. Further, because I work in California and perform work for employees in
California, I am subject to the provisions of a public law SB 1386 and a new law effective January
1, 2005 called AB 1950. SB 1386 requires that if I have an unauthorized person view data
containing sensitive identity theft information, I must inform the affected parties unless authorities
instruct me not to do so, or unless encryption has protected the data in question. However, none
of the major business line application vendors in my industry will go on record supporting EFS or
PGPdrive or any other traditional form of encryption, nor is such encryption native to the
programs.
The new law, AB 1950, requires “reasonable procedures” to ensure that networks are secure
enough to protect identity information for California residents. Gartner Research recommends
that firms follow the guidance in ISO 17799. Keep in mind that this is a California law designed to
protect data of California “residents” no matter where the data resides. Thus, your firm may not
be a California nexus entity, but you would be still covered under this law if your servers contain
data from California residents.
Second, in my environment, as may well be the case in yours, our “bread and butter” line of
business applications come forward from Windows 98 time and exhibit the Windows 98 way of
thinking. Thus, many of my industry’s day-to-day applications require a level of user rights called
“local administrator” on the desktop. This is equivalent to removing all security settings from a
Windows XP or Windows 2000 system. Using tools such as Sysinternals.com’s Regmon and
Filemon, or one from PC Magazine called Inctrl5, to monitor and document permissions needed
on the files, and then set group policies to push down registry settings so the programs have
needed administrative rights without crippling security, takes time and consumes resources to
test and implement. It would be far better to have the application run in “least privilege” in the first
place.
As Aaron Margolis so aptly stated in his blog on the advantages of running as User:
The #1 reason for running as non-admin is to limit your exposure. When you are
an admin, every program you run has unlimited access to your computer. If
malicious or other “undesirable” code finds its way to one of those programs, it
also gains unlimited access. When an exploit runs with admin privileges, its
ability to compromise your system is much greater, its ability to do so without
detection is much greater, and its ability to attack others on your network is
Page
25
greater than it would be with only User privileges. When an exploit requires
administrative privileges (as many do), running it as a User stops it dead.
However, if you’re running as administrator, an exploit can:
•
•
•
•
•
•
•
•
•
•
•
•
install kernel-mode root kits and/or key loggers (which can be close to
impossible to detect)
install and start services
install ActiveX controls, including IE and shell add-ins (common with spy
ware and ad ware)
access data belonging to other users
cause code to run whenever anybody else logs on (including capturing
passwords entered into the Ctrl-Alt-Del logon dialog)
replace OS and other program files with Trojan horses
access LSA Secrets, including other sensitive account information, possibly
including account info for domain accounts
disable/uninstall anti-virus
cover its tracks in the event log
render your machine unbootable
if your account is an administrator on other computers on the network, the
malware gains admin control over those computers as well
and lots more27
In the end, the risks I face in my industry derive not so much from my vendors not supporting
patches, but more from their not supporting users having lesser, that is “user” rights.
Nevertheless, given my fiduciary duty of data protection for my clients, this means that patching
as means of protection makes sense for my industry to remove the vulnerability from the system.
To sum up, once the bulletins become available you need to assign resources to reading and
reviewing them. I will use an historical bulletin to illustrate my methodology. You and your team
need to develop your own methodology of reviewing and testing patches then scheduling patch
deployment. You need a patch-testing matrix a plan for your organization.
Case study—Blaster arrives
Security bulletin Buffer Overrun in RPC Interface Could Allow Code Execution
http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx [aka MSBlaster].
Our first review of this bulletin is that is it rated by Microsoft as a critical bulletin.
Page
26
Graphical view of the bulletin rating system
The web page recapping the bulletin clearly shows that it is affects Windows NT, 2000, XP and
2003. The very first question that I ask is, “Does it affect my network?” If it does, I then review the
rating of the bulletin. Microsoft currently uses a 4-point rating scale. The Microsoft Security
Response Center (MSRC) rates the bulletins based on their impact.28 In general, the MSRC
recommends ALWAYS applying bulletins labeled as critical or important to your systems.
Depending on your network, they recommend that you decide whether to install moderate or low
rated patches. I personally patch anything that moves and, thus, will install any patches in my
network after testing. You must make this decision for your environment. You may decide that
datacenters need only critical and important patches; you may decide that workstations need all
patches. Remember that, for OEM systems, vendors offer patches that they deem critical, but
which are not necessarily security patches. If you scan a test workstation after you patching it by
using Windows Update and find that it reports it is missing a patch, look carefully. That critical
patch may be for a “Dell deemed” critical driver instead.
Microsoft’s rating of the Security Bulletins
Over time, as you use your test network effectively, you will develop an innate sense of which
patches the devices on your networks need depending on the roles those devices play. I strongly
recommend that you set up the ability to RIS or image in your network. Further, your firm will be
much more efficient if you begin to think in terms of the novel concept that states, clearly and
emphatically that employees don’t own the computers that they use.
Because your company owns the computers, controlling what goes on them is your responsibility.
Furthermore, as we noted earlier, you can meet vendor requirements for users to have high rights
levels by adjusting registry entries.
The next step in reviewing the security bulletin gives an indication of the possible threat to a
network:
Page
27
The phrase “run code of attacker’s choice” normally indicates that the attack might be exploitable
remotely. However, there may be mitigation available that is not apparent upon first reviewing the
bulletin.
When is critical not critical?
While bulletin may indicate that the security issue is critical, you still need to review the bulletin for
mitigating factors. Expand the bulletin and review the section called “Technical details.” In this
section, MSRC explains the ranking. Pay close attention to this section as this will determine
when or if you patch. In the “Blaster” example, the “mitigation section” reads as follows:
To exploit this vulnerability, an attacker would need to send a specially formed
request to the remote computer on specific RPC ports.
Mitigating factors:
To exploit this vulnerability, the attacker would require the ability to send a
specially crafted request to port 135, 139, 445 or 593 or any other specifically
configured RPC port on the remote machine. For intranet environments, these
ports would normally be accessible, but for Internet connected machines, a
firewall would normally block these ports. In the case where these ports are not
blocked, or in an intranet configuration, the attacker would not require any
additional privileges.
Best practices recommend blocking all TCP/IP ports not actually in use and most
firewalls including the Windows Internet Connection Firewall (ICF) block those
ports by default. For this reason, most machines attached to the Internet should
have RPC over TCP or UDP blocked. RPC over UDP or TCP is not appropriate
in hostile environments such as the Internet. Robust protocols, including RPC
over HTTP, are suitable for hostile environments.
Our mitigation factors focus on whether protocol ports are open for transmission. Analyze your
systems. A typical network will not have ports 135, 139, 445 and 593 open on the outside
because these are usually trusted network connection ports or file sharing ports. In a typical
business network configuration, these ports would not be open to the outside world. However,
what is YOUR situation? Avoid problems by regularly auditing port settings to ensure that what
you think is open is, indeed, the only thing that is open. Set policies to enforce control and change
management on firewalls in the same manner as you set up change management on operating
systems. Use tools or services to regularly scan and review the exposure on your network.
Consider hiring vulnerability analysis firms to investigate and externally audit these external entry
points. What your “hard shell” looks like from the outside is extremely important. Servers, mail
servers, and data centers “leak” information about what operating systems they run and their
patch levels. Knowing what you look like from the outside helps to identify those entry and threat
points to your network.
Continued analysis of our systems tells us the following:
Role of System
Threat Vector
Exposure
Mandatory Patch needed
Edge device, router, firewall
server, etc
Open and listening port
135, 139, 445 and/or 593
Not open to external
access
Not needed
Because we do not have exposure to the outside, it seems as though we do not need to install
any patches to our systems. It appears that we do not need to take any additional action
whatsoever. Unfortunately, as we all know, the problem was that we are extremely vulnerable on
the inside of our networks. Looking at our networks on the inside, our analysis shows:
Role of System
Threat Vector
Exposure
Mandatory Patch needed
Page
28
Internal server, workstation,
Exchange server
Open and listening port
135, 139, 445 and/or 593
Wide open to inside
exposure
Patch needed
If we could guarantee that all exploits would stay outside our networks, we might not need the
patch inside our networks. However, unless you have network quarantine methodologies, edge
filter or other blocking techniques that you have tested, TCP/IP connections from unvetted
computers regularly connect back to your network. Even in my environment, my network
boundary is not limited to the computers that are physically located in the firm; rather my network
extends to the wireless connection in Starbucks where I am sitting using the Internet to remote
back to my firm.
So what’s the worst thing that can happen?
In the case of the exploit of Security bulletin 03-026, there is quite a bit that can happen. Study
the bulletin carefully:
It is obvious from reading the ‘more information’ section that this is serious. In addition, this
bulletin rates as critical on all platforms. You need to devote resources to testing this patch. The
process affected by this bulletin is something called DCOM RPC interface. So why don’t we just
disable that service or process and go on with life? Because of two things: First, you may have
critical line of business applications that depend on the service, and secondly, many vendors
write their programs using the default enabled services and are unable to tell you if their program
does or does not depend on an affected service. Thus shutting it off may not be an alternative. As
is evident here, DCOM functions as glue in networking:
When you not only recognize which underlying service is vulnerable, but also know which ports
are exploitable, you have clues to the level and type of testing required. Our sample patch
indicates that you will need to maintain normal network connectivity. While I will discuss testing
procedures in a later chapter, for this bulletin, test procedures should cover file transfers and
monitor file transfer speed. In an historic footnote to this bulletin, AutoCAD files29 did indeed
suffer data corruptions because of this security bulletin and a hotfix eventually corrected the
issue.30 Thus, while reviewing the bulletin, you should identify key applications that you will need
to test in your environment. In this instance, those applications that pulled files across the network
would need additional testing time. I strongly urge that you prepare a checklist documenting the
tests, indicating procedures, and identifying applications on which you have tested the patches. If
an issue does arise after the fact, document the affected application, note if the files included in
the patch are re-patched, and assign additional time to test against previously affected programs.
Page
29
Sticking with our 03-26 Security Bulletin as an example, reviewing the affected files shows that
they have changed several times. Now while patches may not affect the same “bits” each time,
even if you don’t have the resources to do a detailed analysis of the patch files, this provides
evidence that you may need resources to test the applications in your firm that had issues with
those files whenever you patched them in the past.
Files patched with Security bulletin 03-026 as listed in http://support.microsoft.com/?kbid=823980
Files included in Knowledge base hotfix 824136 (http://support.microsoft.com/?kbid=824136) that
patched a corruption issue in AutoCAD files
Files included in Security bulletin 03-039 that also affected RPC and replaced 03-026 as listed in
http://support.microsoft.com/?kbid=824146 31
This is just a sample of the impact of this one bulletin. For every security bulletin, you need to
assign a team to review, identify your exposure, identify those services and processes most
affected, and test on your standard server and workstation images.
Resources for Threat Assessment
Many of you may be thinking, “Well this is a fine analysis for an historical patch” but how do I find
guidance for current issues? Among the chief resources for Microsoft patches is the “Threats and
Countermeasures Guide: Security settings in Windows 2003 and Windows XP.” Find this at
http://www.microsoft.com/downloads/details.aspx?FamilyId=1B6ACF93-147A-4481-9346F93A4081EEA8&displaylang=en.
Other resources include:
Page
30
•
•
•
•
NSA Security Configuration Guides:
http://www.nsa.gov/snac/index.cfm?MenuID=scg10.3.1
Center for Internet Security Guides: http://www.cisecurity.org/
Open Web Application Security Project: http://www.owasp.org/index.html
IANA master listing of ports used on the Internet: http://www.iana.org/assignments/portnumbers
If you have not yet inventoried your network, take this opportunity to document what services are
running and could be possible threats. One source for this can be a WMI script that pulls this
information from the server. Find an example at SYDI - Script Your Documentation Instantly:
http://sydi.sourceforge.net/
Remember:
• You cannot protect what you do not understand.
• You cannot protect what you do not know you have.
• You cannot control what you do not access.
Additional testing procedures
Participate in patch communities to find additional resources regarding patch issues. Posters to
resources such as the Patch Management32 listserve, Patch Talk33 and even newsgroups34
regularly share information and patching experiences. If you have few resources to devote to
patch testing, your next best option is to find a resource for your industry. For those industries
where you are restricted from patching, several online resources35 may provide guidance for
properly setting up network segmentation, or may point to other resources for requesting vendors
to test patches themselves. Sharing information can be a very effective resource for patch issues
if you have minimal resources for testing.
Testing procedures may include effectiveness testing
If your testing process includes a check off for independent verification of patch effectiveness,
your next task is identifying resources to obtain code you can use for testing.
Testing patch effectiveness with test code involves using “live exploit” code.
Be sure to do this in a segmented test lab that is nowhere near a production
system. Ensure that you have an extreme air gap, meaning no electrons can
go between this test network and your production network, and that you do
not inadvertently introduce destructive code into your production network.
Use this approach only if you have trained staff, a controlled environment,
and the proper resources. This is not a trivial testing process; do not begin it
without taking proper prevention and exercising caution. For example,
FSecure uses an RF shielded lab when analyzing malware. This is
discussed in a blog post at http://www.f-secure.com/weblog/#00000373.
Typically, there are two resources for test code or exploit code. The first can be the security
bulletin itself. Included in the bulletin are references to the CVE-Mitre vulnerability database. At
times, this database points back to discussion of exploit code on the web.
Our example bulletin of 03-026 contains a link pointing to the following:
Page
31
The hyperlink, in turn, points to the web site at http://www.cve.mitre.org/cgibin/cvename.cgi?name=CAN-2003-0352 that includes a section of references. Included in this
section are web sites and links. Following these links, we find that some are mere
announcements on open vulnerability listserves, but some postings provide more detailed
analyses and include code in their postings such as the one found at
http://marc.theaimsgroup.com/?l=bugtraq&m=105914789527294&w=2.
Examples of ways people share on these listserves include Jeremiah Cornelius’ analysis of the
timeline between patch distribution and the first signs of exploit code on the web posted to
http://lists.netsys.com/pipermail/full-disclosure/2003-August/009278.html. Ben Jurry posted the
first proof of concept code, and H.D. Moore of the Metaspoit project36 refined the code. Ready-torun versions of the exploit were on the web by July 26 at http://lists.netsys.com/pipermail/fulldisclosure/2003-July/007103.html. While the process is not for the faint of heart, if your
environment requires such testing, it is not long between the patch’s development37 and the
exploit community's attempts to reverse engineer and develop exploit code.
MITRE Database pointing to additional resources
There are also web sites, listserves, and RSS feeds that specialize in exploit notification.
Examples include:
•
•
•
•
•
•
•
•
Full Disclosure mailing list: http://lists.netsys.com/mailman/listinfo/full-disclosure (no
moderation is done on this list so you have to weed through postings to find the exploits)
Bugtraq mailing list: http://www.securityfocus.com/archive/1
K-otik RSS feed of their vulnerability list http://www.k-otik.com/exploits.xml
K-otik - Alertes Sécurité: http://www.k-otik.com/mailing (you don’t need to read French to
track vulnerabilities)
Security Focus Vulnerability RSS feed:
http://www.securityfocus.com/rss/vulnerabilities.xml
Security focus mailing list for potential vulnerabilities:
http://www.securityfocus.com/archive/82
Packet Storm Advisories: http://www.packetstormsecurity.org/advisories100.html
Vulnwatch: http://www.vulnwatch.org/subscribe.html
Page
32
A safe way to confirm that a patch has properly deployed into your test scenario is merely to
review the location in the registry. On the other hand, if you use a patch installer program that
provides verification, rerun the patch tool to ensure deployment. You might use the Microsoft
Baseline Security tool (http://www.microsoft.com/technet/security/tools/mbsehome.mspx) for this
or, in our sample bulletin case, open regedit and review the following regkey for installation
confirmation (sample of Windows 2003 reg key entry):
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Updates\Windows Server 2003\SP1\KB823980
Also pay attention to requirements to reboot the system before the patch takes effect. There is an
effort within Microsoft to keep patches from requiring reboots, but as of now, this process is not
yet complete.38
Information about restarting in Security Bulletin 03-026
If you remember the history of Security bulletin 03-026, you will remember that some companies
believed that they had patched. However, because they had not rebooted they failed to protect
their systems. In the case of 03-026, additional, built-in tools ensured proper file replacement.
You may also want to use tools located at the Sysinternals39 web site such as Listdll to perform
additional testing.
Standardize your office
One way to simplify the patching process involves standardization of images. Using a standard
image40 to deploy your systems greatly increases the supportability and predictability of your
network. Documenting the approved software in your network and allowing installation of that
software only helps deliver expected test and rollout results. Change management means that
you approve ALL changes.
A recent post to the Patch Management listserve included a summary by Dominic White41 of Sun
Microsystems’ suggestions for patching.42 The guidance is universal:
•
•
•
•
•
Analyze the need to apply patches or update your software based on risk, cost,
availability, and timing
Minimize change to your environment whenever possible
Address SunAlertSM notifications (Security bulletins) and other critical issues as soon as
possible
Making other changes to your environment only to address known problems
Keep your environment as current as appropriate for your business and application
needs
To summarize—assign appropriate resources to test patches. This is crucial to maintain up time,
and ensure that you manage issues that may arise. Patch testing is just one part of the change
management process.
Testing processes
So how do you test a patch? The most basic test process involves applying the patch to samples
of your normal environment of workstations, servers, data servers, and the like. This is where
standardization of computer images is especially handy. The more you standardize, the faster
Page
33
you can test patches against your images and roll them out to your workstations and servers. By
now, you should be visualizing a patching checklist for your own environment:
Testing plans
Identify the operating system subject to testing.
Identify the service pack level.
Identify the hotfixes installed on your systems (if in addition to security fixes).
Identify critical third party applications.
Identify third party applications that have had patching historically.
Identify those files used in patches that may have causes issues in the past. Are the included in this current
patch? Assign testing resources appropriately.
Study the bulletin to determine if you can uninstall the patch. If not, determine if additional resources for
testing or imaging need to be in place before approving the patch.
Test the installation of the patch both manually and via your automated patch technology.
Can you uninstall the patch using add/remove program or your patch tool?
Review the processes of your line of business applications. Are they performing as expected?
Attempt to replicate a production environment using imaged data. Having an exact image provides the best
testing bed.
Set up performance and monitoring tools to review your testing machines43 such as PerfMon, tools from
Sysinternals44 and review all log files.
Confirm the installation of the patches via registry review or other means.
(OPTIONAL) Confirm the effectiveness of the patch using testing code.
Follow any additional procedures your situation requires.
Approve the patch for release.
PREPARE BACKUPS.
The Infraguard Technology Risk checklist also includes the following:
•
•
•
•
•
•
•
•
•
•
•
When applying a patch to any system vulnerability, verify the integrity of, and test for the
proper functioning of the patch
Verify that the patch will not negatively affect or alter other system configurations
Test patches on test beds before being released into the network
Backup your system before applying patches
Conduct another vulnerability test after you apply a patch
Keep a log file of any system changes and updates
Prioritize patches prioritized
Disseminate patch update information among the organization's local systems
administrators
Add timetables to patch potential vulnerabilities
Require that external partners deploy all non-critical patches within 30 days
Require that external partners deploy critical patches to servers and clients within 48
hours45
Page
34
Think about building your own test policy. In your perfect world, you will test patches on
copies/images of your production systems. You will replicate your user load and your normal
processes. You will identify your critical applications and related issues. That is the perfect world.
Do all this and you will be ready for our next step of rolling out patches to the network in a
controlled fashion—and then we will talk about reality.
Page
35
Chapter 4 – Are we ready to patch yet?
T
here is one key point that we need to emphasize. Even though we try to replicate our
systems as best as we can, there will be times when we cannot perform an exact test to
find all the expected issues that may arise in connection with deploying a patch. Thus, it is
critical that you have backups and system images before you begin to roll out patches.
The following are a few of the available backup and imaging solutions for Windows and Linux
environments:
• Symantec Ghost
• Acronis True Image
• Veritas
• Ultrabac
• Native NT backup
• UniTrends
• Arkeia
Remember evaluate your network from a risk analysis standpoint, even if you have a proper
backup or image of those stations you are patching. What systems are the most at risk for the
patch in question? Carefully review security bulletins and assign patch priority based on the role a
system plays in your network. Look again at patch guidelines such as Security bulletin 04-01146
for the Sasser worm. Security bulletin 04-01147 was a comprehensive bulletin that patched
several security issues. It covered vulnerabilities that exposed edge devices to denial of service
issues. “Chatter” on the listserves indicated that systems directly attached to the Internet and at
the edge of a network needed immediate patching. For this reason, some firms, under certain
circumstances, take the calculated risk of patching edge systems without pre-testing.
Patch “at risk” machines first
Identify regions in your network that are most at risk and assign zones. This is similar to the
“Organizational Units” concept in the Active Directory setup. Zone out and patch servers
separately from workstations, and build patch templates for each zone to scan and patch each
zone separately. Internet Explorer, Outlook Express, or Windows Media patches may be more
critical on desktops than they are for servers, as no one surfs or reads e-mail from server
machines. Furthermore, in Windows 2003, the Internet Explorer enhanced lockdown ensures that
most Internet Explorer patches are not highly critical. The reverse is true for desktops and
machines used by information workers who surf for research purposes. Here there is a higher
priority on Internet Explorer patches. Next, prepare templates for machines that you can patch
more quickly than other machines due to their role in the firm or because they do not require prepatch backup procedures. Develop zones so that you can patch different machines in different
ways.
For those devices that do not permanently attach to your network, you may need a combination
agent/agentless patch tool. We will address this in detail later, but the type of patch tool you have
may dictate how you process the patches. Whether you manually script it, use Software Update
Services, or use one of several available Patch Management software tools, your ability to control
the patch process dramatically increases the more you automate this process. Additionally
automation documents the change process and audits the installation of the patch process.
Even after you have properly tested, consider doing a controlled test roll out to designated
testers. This is particularly true for different types of patches such as a service pack that contains
many more code changes than a security patch does. After your test team has approved the
service patch for release, consider a staggered rollout. A testing environment can go only so far.
It can never truly anticipate all of the issues that may arise in a production network. In testing of
Windows XP SP2, we found that, even though we had no issues during testing, once deployment
began, we saw issues with desktops that had digital graphics cards installed. We needed to track
down an updated digital video driver. We had not deployed to the exact same hardware on which
Page
36
we had tested and we encountered unforeseen issues during the rollout. Your pre-rollout testers
should be computer literate and capable of clearly reporting issues to your help desk team for
resolution. Once you begin patch rollout, notify employees so that they can be sensitive to the
pre- and post-patch issues they may see.
Screen shot of one available patch program, HFNetChkPro
Use patch tools to confirm compliance with policies
Use patch management tools to augment your firm’s policies and to audit the effectiveness of
your external vendors. Additionally you may have contractual lease agreements with external
vendors to manage systems. As part of these agreements, you can request that either you or
they monitor the status of patches.
Further, use your patch management best practices process to scan your systems periodically
between patch periods to ensure that users have not added vulnerable machines to your network.
Build this procedure into your agreements, and make it part of your normal control process so
that you maintain proper control of your company’s systems. Clarify polices regarding patch
responsibility. As we have stated earlier, if your vendors will not allow you to patch, you must
scan and monitor your system so you can isolate, protect, and defend.
Page
37
Ecora Software graphical view of patch status
Capture feedback
Every organization has or should have formal or informal help desk capturing tools48 for tracking
issues relating to the patches you roll out. Ensuring that you quickly identify issues and work with
your vendors is extremely important. We have seen cases where a patch application changed the
way in which a third party program interacted with a system. When this happens, begin working
with the vendors. In the case of Microsoft patches, any issue caused by a security patch is a free
support49 call. Red Hat tracks issues in their bug database.50 It is also wise to build up tech
support relationships with antivirus vendors, firewall vendors, and line of business application
vendors. Determine if they have e-mail notification services for issues with security patches. With
excellent cooperation and communication, we can ensure the smoothest release process.
Microsoft has put in place processes to assist in testing their patches prior to release. As one
industry observer has noted, “Microsoft made good on the promise to beta test security patches
with a limited number of customers and partners. By testing the patches in real-world
environments, this program succeeds in increasing the quality of final patches.”51 A similar
feedback process in your environment contributes to patch process success.
Enforcement
We assume that your corporate policy mandates that all machines should have patches installed;
however, without enforcement of the policy, you systems remain exposed. Here is the rule:
“Policy First; Technology Second”. In the case of security policies regarding the level of patches
on systems, you can find guidance on setting and developing your policies on various web sites.
The SANS.org52 Security policy page highlights the following sites:
•
http://www.security.kirion.net/securitypolicy/
Page
38
•
•
•
•
•
•
•
•
•
http://www.network-and-it-security-policies.com/
http://iatservices.missouri.edu/security/
http://www.utoronto.ca/security/policies.html
http://irm.cit.nih.gov/security/sec_policy.html
http://w3.arizona.edu/~security/pandp.htm
http://secinf.net/ipolicye.html
http://cio.berkeley.edu/policies.html
http://www.ruskwig.com/security_policies.htm
http://razor.bindview.com/publish/presentations/InfoCarePart2.html
There are also numerous examples of security policies on the SANS.org web page. The key
elements you can find from these documents include:
From the Server Security policy53:
•
Install the most recent security patches as soon as practical, the only exception being
when immediate application would interfere with business requirements.
From the Acquisition policy54:
•
•
Install a standard <Company Name> image on all hosts—servers, desktops, and laptops.
Business critical production servers that cannot be replaced or re-imaged must be
audited and a waiver granted by InfoSec.
From the Audit policy55:
•
Scanning period: <Company Name> and <Internal or External Audit Name> Scanning
Team shall identify in writing the allowable dates for scans to take place.
From the VPN connection policy56:
•
All computers connected to <Company Name> internal networks via VPN or any other
technology must use the most up-to-date anti-virus software that is the corporate
standard (provide URL to this software); this includes personal computers
Sample Test Lab Security policy
Given how critical it is to your test network, we provide an entire Test Lab Security policy
referenced and available from the SANS.org site. It points out the importance of separating the
Lab from the rest of the network. We highlight the test lab security policy in particular, as it is
imperative that you keep your test lab from infecting the rest of your production network. Isolate it
accordingly.
Internal Lab Security Policy57
1.0 Purpose
This policy establishes information security requirements for <Company Name>
labs to ensure that <Company Name> confidential information and technologies
are not compromised, and that production services and other <Company Name>
interests are protected from lab activities.
2.0 Scope
This policy applies to all internally connected labs, <Company Name> employees
and third parties who access <Company Name>'s labs. All existing and future
equipment, which fall under the scope of this policy, must be configured
Page
39
according to the referenced documents. DMZ Labs and stand-alone, air-gapped
labs are exempt from this policy. DMZ labs must comply with the DMZ Lab
Security Policy.
3.1 Ownership Responsibilities
1. Lab owning organizations are responsible for assigning lab managers, a
point of contact (POC), and a back-up POC for each lab. Lab owners must
maintain up-to-date POC information with InfoSec and the Corporate
Enterprise Management Team. Lab managers or their backup must be
available around-the-clock for emergencies, otherwise actions will be taken
without their involvement.
2. Lab managers are responsible for the security of their labs and the lab's
impact on the corporate production network and any other networks. Lab
managers are responsible for adherence to this policy and associated
processes. Where policies and procedures are undefined lab managers must
do their best to safeguard <Company Name> from security vulnerabilities.
3. Lab managers are responsible for the lab's compliance with all <Company
Name> security policies. The following are particularly important: Password
Policy for networking devices and hosts, Wireless Security Policy, Anti-Virus
Policy, and physical security.
4. Lab managers are responsible for controlling lab access. Access to any
given lab will be granted by the lab manager or designee only to those
individuals with an immediate business need within the lab, either short-term
or as defined by their ongoing job function. This includes continually
monitoring the access list to ensure that those who no longer require access
to the lab have their access terminated.
5. The Network Support Organization must maintain a firewall device
between the corporate production network and all lab equipment.
6. The Network Support Organization and/or InfoSec reserve the right to
interrupt lab connections that affect the corporate production network
negatively or pose a security risk.
7. The Network Support Organization must record all lab IP addresses, which
are routed within <Company Name> networks, in Enterprise Address
Management database along with current contact information for that lab.
8. Any lab that wants to add an external connection must provide a diagram
and documentation to InfoSec with business justification, the equipment, and
the IP address space information. InfoSec will review for security concerns
and must approve before such connections are implemented.
9. All user passwords must comply with <Company Name>'s Password
Policy. In addition, individual user accounts on any lab device must be
deleted when no longer authorized within three (3) days. Group account
passwords on lab computers (UNIX, windows, etc) must be changed
quarterly (once every 3 months). For any lab device that contains <Company
Name> proprietary information, group account passwords must be changed
within three (3) days following a change in the group’s membership.
10. No lab shall provide production services. Production services are defined as
ongoing and shared business critical services that generate revenue streams
Page
40
or provide customer capabilities. These should be managed by a <proper
support> organization.
11. InfoSec will address non-compliance waiver requests on a case-by-case
basis and approve waivers if justified.
3.2 General Configuration Requirements
1. All traffic between the corporate production and the lab network must go
through a Network Support Organization maintained firewall. Lab network
devices (including wireless) must not cross-connect the lab and production
networks.
2. Original firewall configurations and any changes thereto must be reviewed
and approved by InfoSec. InfoSec may require security improvements as
needed.
3. Labs are prohibited from engaging in port scanning, network auto-discovery,
traffic spamming/flooding, and other similar activities that negatively impact
the corporate network and/or non-<Company Name> networks. These
activities must be restricted within the lab.
4. Traffic between production networks and lab networks, as well as traffic
between separate lab networks, is permitted based on business needs and
as long as the traffic does not negatively affect other networks. Labs must
not advertise network services that may compromise production network
services or put lab confidential information at risk.
5. InfoSec reserves the right to audit all lab-related data and administration
processes at any time, including but not limited to, inbound and outbound
packets, firewalls and network peripherals.
6. Lab-owned gateway devices are required to comply with all <Company
Name> product security advisories and must authenticate against the
Corporate Authentication servers.
7. The enable password for all lab owned gateway devices must be different
from all other equipment passwords in the lab. The password must be in
accordance with <Company Name>'s Password Policy. The password will
only be provided to those who are authorized to administer the lab network.
8. In labs where non-<Company Name> personnel have physical access (e.g.,
training labs), direct connectivity to the corporate production network is not
allowed. Additionally, no <Company Name> confidential information can
reside on any computer equipment in these labs. Connectivity for authorized
personnel from these labs can be allowed to the corporate production
network only if authenticated against the Corporate Authentication servers,
temporary access lists (lock and key), SSH, client VPNs, or similar
technology approved by InfoSec.
9. Infrastructure devices (e.g. IP Phones) needing corporate network
connectivity must adhere to the Open Areas Policy.
10. All lab external connection requests must be reviewed and approved by
InfoSec. Analog or ISDN lines must be configured to accept only trusted call
numbers. Strong passwords must be used for authentication.
Page
41
11. All lab networks with external connections must not be connected to
<Company Name> corporate production network or any other internal
network directly or via a wireless connection, or via any other form of
computing equipment. A waiver from InfoSec is required where air gapping is
not possible (e.g., Partner Connections to third party networks).
4.0 Enforcement
Any employee found to have violated this policy may be subject to disciplinary
action, up to and including termination of employment.
5.0 Definitions
1. Internal - A lab that is within <Company Name>'s corporate firewall and
connected to <Company Name>'s corporate production network
2. Network Support Organization - Any InfoSec approved <Company Name>
support organization that manages the networking of non-lab networks.
3. Lab Manager - The individual responsible for all lab activities and personnel
4. Lab - A Lab is any non-production environment, intended specifically for
developing, demonstrating, training and/or testing of a product.
5. External Connections (also known as DMZ) - External connections include
(but not limited to) third-party data network-to-network, analog and ISDN data
lines, or any other Telco data lines.
6. Lab Owned Gateway Device - A lab owned gateway device is the lab
device that connects the lab network to the rest of <Company Name>
network. All traffic between the lab and the corporate production network
must pass through the lab owned gateway device unless approved by
InfoSec.
7. Telco - A Telco is the equivalent to a service provider. Telcos offer network
connectivity, e.g., T1, T3, OC3, OC12 or DSL. Telcos are sometimes referred
to as "baby bells", although Sprint and AT&T are also considered Telcos.
Telco interfaces include BRI, or Basic Rate Interface - a structure commonly
used for ISDN service, and PRI, Primary Rate Interface - a structure for
voice/dial-up service.
8. Traffic - Mass volume of unauthorized and/or unsolicited network
Spamming/Flooding traffic
9. Firewall - A device that controls access between networks; it can be PIX,
that is, a router with access control lists or similar security devices approved
by InfoSec.
10. Extranet - Connections between third parties that require access to
connections non-public <Company Name> resources, as defined in InfoSec's
extranet policy (link).
11. DMZ (De-Militarized Zone) - This describes network that exists outside of
primary corporate firewalls, but are still under <Company Name>
administrative control.
6.0 Revision History
Technical enforcement
We have our policies, our scripts, or patch tool, but how to we enforce policy compliance? Within
the world of technology vendors, we are starting to see corporations adding tools to normal
network services. Both CISCO58 and Microsoft59 currently provide a Network Quarantine feature
to the systems that they offer. The offerings are both Server based and IP based approaches.
This technology is primarily for VPN connections at this time and is estimated to include normal
internal network traffic by 2005. The concept is that a computer system must pass a predefined
standard before they connect to the environment. Forrester recommends that you evaluate and
explore mixing technologies, however recent news has indicated that the two major players,
Microsoft and Cisco, may not be working with one another and you may have to choose.60
Page
42
61
Graphical view of Microsoft Network Quarantine Services
Pete Boden, Director of US IT Corporate Security at Microsoft, used a TechNet Radio62 session
to discuss ways of ensuring that systems match the policies. At Microsoft, they perform tests that
Make sure the right version of Connection Manager is being used; do that to
make sure that all of the subsequent checks are in place and are current. Then
check for Windows XP Professional; that’s our remote-client-connection standard
OS, so everybody who connects remotely to our environment should be running
Windows XP Professional. Then we check that the Internet connection firewall is
enabled, that Internet connection sharing is disabled, and that the antivirus
product is loaded and all of the signature definitions are updated. After making
that connection, we launch a full compliance scan of the machine from our
network, which essentially runs through the rest of the patch and configuration
compliance checks. We make sure that the machine is fully patched; we make
sure that it’s configured correctly, and if it’s not, we deal directly with the asset
owner, the owner of that PC, to make sure that those checks or that those
standards are then met.
The compliance process starts with discovery and assessment. That means, first
of all we have to become aware that vulnerability exists. Typically, we use
mechanisms like the Microsoft Security Response Center, at least for Microsoft
vulnerabilities. Once we become aware of a vulnerability, we rate its severity to
determine how critical that vulnerability is and what we need to do to remediate.
If it is a critical vulnerability, we set a deadline for compliance, which is simply a
date by which every system on our network needs to comply with our patch
standard. The next step evaluates compliance across the network. As we set
those standards, and as we set the compliance deadline, we’re performing
baseline scans across our environment to see how compliant we already are. As
vulnerability information is released to our client base, many clients go out and
update themselves. They are allowed to update their own system and make sure
it’s compliant. At the same time we test to make sure that the patch is of high
enough quality to deploy, so we provide it to a very select group of application
owners to make sure that they can install that patch, run their business
appropriately, and uninstall it if they need to. Then we get to the deployment
phase, and this is the phase where we’re doing an enormous amount of
communication with our business systems owners and clients on the network to
make sure that they understand when they need to be patched by and what the
consequences are for noncompliance. Once we reach that patch deadline, we
drop into what we call the enforcement phase. Once we get to that phase, when
Page
43
we identify a vulnerable system we take one of two measures. We either forcibly
patch or configure that machine or we remove it from the network.
Let’s stress this point again. These computers are your company assets; they do not belong to
the employees. Thus, setting policies and compliance is critical to maintain control.
Set the policy to make it easy to update and comply with
On the patch management listserve, people often ask for a template of all needed patches for a
workstation or server. What is disconcerting in this question is that there is a desire to rely only on
human interpretation of what is currently installed on a system. It’s much better to have a
combination of human expectation based on a review of the published bulletins, and a
reinforcement of this awareness by using a patch tool to confirm that “yes, you need to patch for
this.” While this book is sponsored by a patch management software solution vendor, we cannot
stress this point enough so we will make it loud and clear: If you rely solely on human
judgment for patching your systems, your risks are significantly increased and your rate
of failure is extremely high.
Whether you use free patch tools or develop your own scanning methodology, relying solely on
judgment and not following up with some sort of tool or technique to confirm your patch status is
extremely foolhardy. Patch and then CONFIRM the application and effectiveness of the patch. In
the windows environment confirm by checking registry keys or current, in-use DLLs to ensure
patch status. Utilities such as tlist, ListDLLs63 or Process Explorer64 assist in this process.
Stuff happens
Even with all of our testing, things happen. Therefore, ensure that you have good vendor
relationships. There is nothing so frustrating than the “ping pong” effect: Vendor A blames an
issue on Vendor B; Vendor B blames Vendor A. The more a specific vendor depends on your
business, the more power you have. While migrating to different vendors is not easy, the fact that
we are in an increasingly competitive marketplace helps greatly. On listserves and newsgroups,
we repeatedly see people post issues that they, clearly, could move to the vendor for resolution.
Repeatedly they are reluctant to do so. If cost is an issue, remember that calls to resolve issues
caused by a Microsoft security patch are free. When contacting any vendor, gather the following
information:
•
•
•
Details about the computer exhibiting the problem
o Version installed
o Service packs applied
o Affected application
Version
service pack
o Other applications on the box
o Antivirus software, version
o Processor
o Memory
o Hotfixes Installed
Details about other computers (a client machine, for example), if applicable
o Operating system and service pack
o Applications
o Antivirus software, version
Details about the network, if applicable
o Internet connection type
o Number of NICs in the server
o Firewall/proxy
o Network topology particulars
Page
44
•
Details about the issue
o Full, complete error message wording or screenshot
o Timing of first occurrence
o Changes made around that time
o Steps taken to reproduce the error, if any
o Additional information you think might be helpful
There is a cartoon showing Bart Simpson writing on the blackboard, “I will search Google before
asking dumb questions.”65 You would be surprised how many people do not know the power of
the search engine. You can begin your error investigation process by using one of the greatest
diagnostic tools around: A search engine such as Google. Launch the Google search engine66
and put in your "exact" error message. You may be surprised that someone else had your same
problem and someone else has helped him or her find the answer. If that does not work, try
searching on Google Groups67, which is the search engine of the Usenet. To narrow your search
further, click on “Advanced” and limit your search to microsoft.com or redhat.com or the
appropriate vendor’s domain.
For narrowing down Usenet postings, click on advanced and include a wildcard, as in Microsoft.*,
which will search on the public newsgroups on the Microsoft domain. Lastly, after you review the
Event viewer, the web site of www.eventid.net is invaluable for quick resolution on security or
even non-security issues. Subscribe to the service. Those red stop signs in your event viewer and
the resulting code can point you in the direction of resolution.
Be willing to gather information
Without data, your vendor is dumb and blind to the issue you are seeing. If your system crashes
and asks to send a “Dr. Watson” dump to Microsoft, for example, send it. The information sent to
Microsoft or any vendor assists them greatly in pro-actively preventing issues. You may find that
doing a Dr. Watson error gets you a link to additional information for resolution. Next, have on
hand the tools that Microsoft Product Support or other vendors use in diagnosing issues. Tools
such at the PSS support tools68 allow engineers to more easily see what is going on. The support
engineer may also want you to use debugging programs69 to prepare a dump file for analysis.
Use additional tools from the Server resource kits70 as needed. The more information you provide
the faster and better will be the resolution. Finally, when you call in for support to any vendor,
make sure that you get to the right division or department.
Online communities offer another way to compare patch experiences. Phrase your subject line
carefully when sending a question and describe the steps that you have taken. Include the
relevant information to make the resolution process faster. Find additional guidance in asking
effective questions the right way at http://www.catb.org/~esr/faqs/smart-questions.html
Remember: There are no dumb questions. Thinking it is a dumb question is just dumb
because you will not find an answer until you ask the question. Always ask, as someone else
probably has the same question that you have.
Change Management follow-up
You’ve patched and you’ve enforced the policy to ensure that everyone patches. So now you are
done, right? Wrong. First, you need a change review process to summarize and gather those
items that testing missed. Can you cover an emerging issue in future testing procedures? Can
you better follow up with vendors and personnel? Did the patch affect one application in
particular? Use this review process to learn and make the next patch process easier.
Page
45
Microsoft’s representation of the Change Management process
Determine which tools did the best job of recreating your network. If there were issues with
patches, what backup or imaging procedures worked the best? Did you ensure that during this
process, all appropriate change control mechanisms were in place? Use this process to learn and
refine your procedures as needed on an ongoing basis.
So I can’t patch, now what do I do?
There will be times when patching is either not an option or, in the case of zero day exploits
wherein the exploit is released to the web before a patch has been built. You might find that you
need to take alternative actions. Nevertheless, be sure to analyze carefully the true risk. Many
businesses that didn’t patch for 03-026 erroneously believed that they were safe because they
had a firewall; however, all MSBlaster (03-026) needed to shut down a business was a remote
connection in from an infected remote client. The recent addition of the XP SP2 firewall71
controlled from the INSIDE of a company helps limit internal exposure.
If contractual arrangements restrict you from patching, consider “air gaps” or other devices to limit
and restrict TCP/IP traffic. Vlans and virtual machines may not be enough to limit such traffic as
switches can be vulnerable devices as well. You can also use IPSec,72 which is “an
implementation of the Internet Engineering Task Force’s Internet Protocol security (IPSec).
IPSec, which is also included in Windows 2000 and Windows XP, provides network managers
with a key line of defense in protecting their networks. IPSec exists below the Transport layer, so
applications transparently inherit its security services. IPSec provides the protections of data
Page
46
integrity, data origin authentication, data confidentiality, and replay protection without having to
upgrade applications or train users.”
IPSec provides defense-in-depth against vulnerabilities in upper-layer protocols and applications by
protecting upper layer protocols, services, and applications. With IPSec enabled, initial packets
access an application or service running on a server only if they meet the allowed policy. For
example, attempts to disable applications or services on servers must first penetrate the IPSec
protection. In addition, because packets will not be passed to the application or service until trust has
been established through IPSec authentication and the configured protection on packets for the
application or service is protected.
Use IPSec policy to restrict access to servers. You can configure a server to accept specific types
of traffic only. For example, you can configure an e-mail server to accept only secured e-mail
traffic from client computers. The e-mail server discards all other traffic from client computers.
What is the best way to find alternative to patching? Turn first to the security bulletins themselves.
Using our Security bulletin 03-02673 as an example, refer to the technical information section.
Scroll to the “Mitigating factors” area located in each bulletin. In this case, mitigation involved
blocking the affected ports. This is easy to do on a stand-alone machine, but hard to do in a
network.
The story of two patches
Two patches serve as examples of “major events” for IT professionals. Both were highly critical,
many administrators pushed both patches out to their network EXTREMELY fast. Moreover, while
one turned into a network event, the other seemed not quite the Internet threat that it initially
appeared to be.
Let’s look first at the MSBlaster security bulletin 03-02674 to review why it became an event:
Security bulletin 03-026 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980)
MSBLAST
Issue
Impact of vulnerability
Result
Run code of attackers choice
Severity rating
Critical
Threat vectors
File and printing ports
Comment
“Run code” is translation for “oh this
is going to be bad”
Also critical on ALL platforms
[sometimes Windows 2003 has
mitigation issues when there are
security issues for Internet Explorer
due to the IE enhanced lockdown”
Network administrators thought they
had more control over the
parameters of their network. We
don’t. Plan accordingly.
Regarding Blaster, we knew that our firewall protected us on the outside. What we did not take
into account was how interconnected we are. The vulnerability for 03-026 spawned a worm called
MSBlaster. This worm got inside of our networks because of VPN and other remote connections.
It took down firms and put their servers and workstations offline. Even today, we remain at risk on
the inside; we cannot depend on our network perimeters alone. In-depth defense through
Page
47
compliance with patch policies would have prevented MSBlaster. Now let us look at our second
sample patch:75
Security bulletin 04-011 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980)
SASSER
Issue
Impact of vulnerability
Severity rating
Result
Remote code execution
Critical
Threat vectors
Primarily File and printing ports
Comment
Hmmm… this also sounds bad
But here’s the difference – NOT
critical on Windows 2003 servers
Primarily affected computers without
up to date firewalls and antivirus
Of all the issues addressed in this bulletin, the malware targeted only the LSASS service. The
mitigation of this one was clear:
An anonymous user can remotely attack Windows 2000 and Windows XP only.
While Windows Server 2003 and Windows XP 64-Bit Edition Version 2003
contain the vulnerability, only a local administrator could exploit it.
Given reviews of security issues, both patches deserved very quick installation, especially when
administrators compared notes and looked at the threat vectors. Many thought that the Sasser
worm would have exploited the SSL vulnerability instead of the LSASS service. Nevertheless,
each case highlights the same thing: The time between patch, exploit and event is shrinking.
The coming threats
Patching is just one source of protection. Forthcoming network protection technologies are just
the latest in the arsenal of tools that we must use to protect ourselves. Remember: it’s not just
about having a tool for patching; it’s about controlling the patching and enforcing patching best
practices. If you do not control your systems, they will control you.
We will complete our technical discussion of patches by looking at what lies ahead. A Microsoft
article76 seeks to:
• Establish a single set of patch related terms and mandate their use in all documentation
related to patches as described in knowledge base article 824684
• Establish the role, format, and content of knowledge base articles published in
conjunction with patches. The KB article will focus on installing the patch, while the
bulletin will focus on the vulnerability. as described in knowledge base article 824689
• Establish a common naming convention for all patch patches, providing information such
as the product the patch applies to, the language the patch was developed for, as
described in knowledge base article 824685
• Establish the standard that all patches use a properties page as discussed in knowledge
base article 824686
• Establish a single consistent way of recording a patch in the add/remove programs list.
Microsoft states that they are examining whether it may be appropriate to develop a
repository of information dedicated solely to patches
• Establish a standard such that all Microsoft patches will use either of two installer
technologies:
o Windows Installer [MSI] will be used by patches for applications
o Update.exe will be used by Windows operating systems and some applications
• Establish a standard of patch size reduction by removing debugging symbols from
patches. {If you receive a hotfix from Microsoft, you will notice for some hotfixes “symbol”
files inside the zipped hotfixes.
• Establish a uniform set of options that installer technologies will support as discussed in
knowledge base 824687
Page
48
•
•
•
Establish a standard for uniform set of return codes and log entries that all patch
installers will use
Establish a standard allowing for uninstalling of all patches
Establish a standard of placing patches in a standard registry location
So what does the future hold for patch technology? Standardization is likely, and it can come
none too soon. As we begin to mix operating systems, applications, and platforms, standards
must emerge and evolve. Dig deep in your applications and work with your vendors to understand
how your systems work, to examine your systems and to understand how to patch them and how
to test the patches. Does this testing replicate your environment? The larger and more complex
your environment, the more ability you have to ask vendors to support and test patches before
you roll them out. Include such specifications in your vendor agreements. Begin to push, slowly at
first, so that your applications support the latest service packs within a reasonable period.
“Demand better security from vendors and hold them responsible. Use what you have, and make
sure you know how to use it properly and effectively”77
Ask your vendors to step up to the plate and support and test patches as well.
The bottom line for your boss
If you are reading this, chances are you are not the CEO or CIO or CFO. You probably do not
have budgetary control and the final say. Refer to the one-page appendix at the end of this
chapter. Copy it, hand it to the people who make the decisions in your firm. Take the time to
calculate the ALE formula. Discuss the impact of how the biggest security impact comes from
laptops that may not get the attention they need to maintain patch level. Take the time to review
the business side of patching to compile arguments you need to get the tools you must have.
Effective patch management is all about good process and good control. If you do not control
your server, your desktops, or your infrastructure; if you do not control who has access to your
network; and if you do not control what comes into and goes out of your network, you don’t
control anything. It is not YOUR network anymore.
The World has changed and computing environments continue to evolve. Companies that may
have considered security a minor compliance issue must now meet security standards. California
already has a law on the books commonly referred to as “SB138678.” This law requires a firm to
notify California residents when an unauthorized party views data sensitive to identity theft. While
this is an admirable law from the standpoint of the person susceptible to identity theft, it is a
potential liability and public relations nightmare. Management often requires ROI calculations on
new processes and purchases, but they often ignore intangible costs. The traditional ALE
equation may not give you the entire cost analysis you need but it may be all you have right now
to make your point.
CIO Magazine79 has predicted that the insurance industry will push for Security ROI and better
industry statistics, but that has yet to happen. We are still moving toward what Kevin Soo Hoo
described as “a place where computer security risks would be ‘actively and effectively
managed.’”80
Do you need to be empowered to take control of your network? Refer to the US CERT research
center for evidence on the survivability of the Internet.81 Consider, also, that Incidents.org, which
track events on the Internet, calculate that a Windows XP workstation can run on the world wide
web without protection for only twenty minutes.82 Ask yourself, if your Help Desk staff cannot get
to that system within that period and your employees control their systems and not your IT staff,
who really controls your network? It is certainly not your IT staff! Where is your weakest link?
Look toward the person who has the ability to download what he or she wants to. Patch
Page
49
management is part of the process of having your IT staff stake a claim for once again controlling
YOUR network.
Page
50
Chapter 5 – Life beyond patching Microsoft
T
he reality of what we protect today includes products beyond those developed and
supported by Microsoft. Third party software we monitor for updates included products from
Apple, Macromedia, Real Networks, Adobe, WinZip, and many others. It even includes
tracking vulnerabilities in antivirus agents. Many firms also track Solaris, RedHat, SuSe, Oracle,
Cisco, and even vulnerabilities in devices and printers from vendors such as Ricoh and HP.
While this text is about the concepts of patch management, one thing is clear. You must clearly
understand what you are patching, know what information resources you have available and, just
as in our extensive discussion of Microsoft patching, arm yourself with an environment and a
team that is able to test, report findings, maintain good relationships with vendors, and manage
Service Level Agreements tied to your goals. While this chapter covers some new ground, you
will find that it will reinforce many of the concepts we have talked about before. Assessment of
your network, testing procedures, policies in place, reporting of patch issues, all of these
processes are identical in any software platform you maintain. The firms and service level
agreements (SLAs) may be different, but the basic concept of “test to ensure you do no harm” is
universal.
Sun Tzu in The Art of War states, “If you know the enemy and know yourself, you need not fear
the result of a hundred battles. If you know yourself but not the enemy, for every victory gained
you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in
every battle.” We would argue that in the battle for better security, we are still working our way
towards knowing ourselves. If you have not yet determined the critical assets you need to defend
and protect, you need to do this analysis first.
Just as in our previous chapters, we assume that you have identified that the software needs a
patch according to the vendor.
No matter what platform, spending resources to patch, putting a business continuity plan in place,
ensuring that patches will not harm systems, arranging for notification of new patches from
vendors, and obtaining patches conveniently are mandatory best practices. Over the Christmas
holidays, a worm exploited vulnerabilities in phpBB (an Open Source Bulletin Board or CMS) and
defaced web sites. Later deemed the “Sanity” worm, it points out that every platform is
susceptible. Irregularly maintained and unpatched systems are the most common denominators
among successful attacks. To prevent attacks we need to patch and to patch we know we must
test.
Recall the basic procedures used to test patches:
•
•
•
•
Best—an exact replica (ghost) of your exact system on the exact same hardware,
applying the patch to the ghosted system and monitoring processes
Better—applying the patch to a mirror of the system, purposely breaking the mirror and
monitoring results
Better—applying the patch to a Vmware or virtual server (virtual PC will suffice with a bit
of a workaround, but it is directly supported on Vmware and Virtual server platforms);
monitoring applications and processes
Applying the patch to a test server/workstation and monitoring the applications and
processes
As always, have a testing matrix and sign off procedures. Our policies and procedures for testing
and approving patches are going to be the same no matter what the platform. Identify what you
have and what you need to protect; remove—do not just disable—what you do not need running;
run the least amount of privileges on files, folders, servers, and workstations to limit exposure;
Page
51
and audit to confirm exposure to risk. These basic concepts apply to any platform you choose to
run.
Patching third party applications
Let us begin this chapter by examining procedures we can use to patch (or actually re-deploy)
sample third party applications on the Microsoft platform. We will then discuss procedures to
patch operating systems such as Linux, Solaris, Oracle, and others.
For many of you, this section, hopefully, will be ‘old hat’ and just a review; but if you have not
used Group Policy to deploy applications, note that patching (or deploying) a third party
application on the Windows platform involves preparing a transform file and utilizing active
directory. On the domain that needs to deploy an application the basic procedure is as follows:
•
•
•
•
Create a distribution point
Create a group policy object for the deployment
Deploy the .msi file from the shared distribution folder as machine-assigned.
You can also deploy the .msi to specific security groups.
Here is an example of deploying Adobe Acrobat reader to all workstations. The concepts mimic
those detailed in the white paper “Using Group Policy to Deploy Windows XP sp2”
(http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/gpdepsp2.mspx) with one
exception. For some applications, you must prepare a file for distribution as an .msi application. In
the case of Adobe, this involves a little “trickery” of pretending to install the program. When you
download the full file, it is actually an executable shell around the msi program. To “expose” the
.msi, you merely go through the motions of having it begin the install and then make sure you
“cancel” at the proper moment.
Continue the installation “past” this screen
Page
52
At this point, select “Cancel”
You will now have on your local workstation system a folder that includes the “expanded files”.
Copy these files up to the server installation point and the distribution point that you will use now
that you have the actual msi file.
Copy these files to your “distribution” point
To create a distribution point83
1. Log on to the server computer as an administrator.
2. Create a shared network folder where you are going to put the Microsoft Windows
Installer package that you want to distribute. This folder is the distribution point for the
software package.
3. Set permissions on the shared network folder to permit access to the distribution
package.
4. Give access permissions to the following: administrators, authenticated users, and
domain users. (Remember that on Windows 2003, the “Everyone” group no longer
includes anonymous users.)
Page
53
Setting permissions for deployment
5. Configure distributed file system (DFS) for the distribution point in larger enterprise
settings. This provides you with more flexibility by ensuring uninterrupted availability of
the distribution point, in case you have to replace the server. In addition, DFS makes it
easier to have distribution points in multiple sites. For more information on DFS review
the information at http://go.microsoft.com/fwlink/?linkid=34229
To create a Group Policy Object (GPO) for deployments84
1. Begin by ensuring that you have a copy of the Group Policy Management Console
installed on a XP sp2 workstation (preferably) that has the recent updated administration
files.
2. Download this console tool from
http://www.microsoft.com/downloads/details.aspx?FamilyId=0A6D4C24-8CBD-4B359272-DD3CBFC81887&displaylang=en. However you can control group policy from a
non-XP sp2 location with a hotfix that will be included in Windows 2003 sp1. Download
the hotfix from http://support.microsoft.com/default.aspx?kbid=842933
3. On an administrative workstation in large environments, log on as the domain
administrator, or in smaller environments, remote desktop to the server and open Group
Policy Management Console (GPMC). This is typically done by typing in “gpmc.msc” in
the start, run box as shown:
Launching the Group Policy editor
4. In the console tree, right-click the domain name in the forest in which you want to create
and link a Group Policy object (GPO).
5. Click Create and Link a GPO Here.
Creating a Group Policy Object
6. In the New GPO dialog box, specify a name for the new GPO, and then click OK.
Preparing the new GPO
Page
54
To edit a GPO for Deployments
1. Right click the new GPO and click Edit.
Preparing to edit the Group Policy Object
2. In the Group Policy Object Editor, click Computer Configuration, click Software Settings,
and then click Software Installation.
3. On the Action menu, point to New, and then click Package.
Preparing to deploy the package [msi]
4. In the Open dialog box, in File name, type the full Universal Naming Convention (UNC)
path of the shared installer package that you want to distribute. Type this path in the
following format: \\ServerName\SharedFolder\Update.msi or
\\ServerIP\SharedFolder\Update.msi. Make sure that you use the UNC path of the shared
installer package.
Ensuring that the package can be retrieved across the domain by using the UNC path
5. Select the Windows Installer package, and then click Open.
6. In the Deploy Software dialog box, click Assigned, and then click OK. The shared installer
package that you selected appears in the right pane of Group Policy Object Editor.
View of the resulting deployment
To target the msi using security filtering
1. In GPMC, double-click Group Policy Objects.
2. Click the GPO to which you want to apply security filtering.
Page
55
Selecting the GPO
3. In the results pane, on the Scope tab, click Add.
4. In “Enter the object name to select,” type the name of the group, user, or computer that
you want to add to the security filter, and then click OK.
5. If Authenticated Users appears in the Security Filtering section of the Scope tab, select
this group and click Remove. This will ensure that only members of the group or groups
you added can receive the settings in this GPO. If you leave this group in, staff logging
into your servers will deploy the application to the servers as well. Set ONLY those
groups that you want to deploy the software to in this window.
Click on the settings tab to view the resulting effects of your deployment and to view the settings
that the group policy will use to deploy the package:
Resulting Group policy screen
It is important to realize that testing a rollout or deployment or patch methodology is very
important. Businesses have suffered dearly when scripts and group policy commands did more
than expected or intended. Prepare a test lab and test the process.
To force the group policy application, from a command prompt at the server, enter gpupdate/force
(for Windows 2003/XP machines)
Page
56
GPupdate/Force
In the case of “scripted” installs, you may want to add additional features to ensure that the
deployment of the software is “silent,” that is, the end user need not interact to apply the software
update. Typically, additional scripting ensures that the install is transparent to the end user. For
additional information and guidance for using “scripted” options, look at the Tech Republic web
page at http://www.techrepublic.com and in the case of Adobe you can further link to
http://techrepublic.com.com/5100-6270-5054963.html. If you have prior versions of Adobe, you
may need to adjust the script to uninstall the prior versions. For other applications, you can
typically ask fellow administrators on the PatchManagement.org listserve
(www.patchmanagement.org) for guidance or Google for recommendations. There is also
additional information for using the Microsoft Command line and the Windows installer here:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/msiexec.mspx
Vendors have their own resources for enterprise deployment. The Adobe web site includes the
following information for Terminal service installations, Citrix installations, and general enterprise
deployment: http://partners.adobe.com/public/developer/acrobat/index_advanced.html Review
your vendor’s technical guidance for enterprise deployment scenarios and suggestions.
Page
57
Patching Linux distributions
A patch in both Windows and Linux products is a band-aid, a fix for a missed security issue or an
interim update prior to a service pack release. In the Linux platforms, it is typically easier to avoid
patching by not installing the application in the first place. Recent movements in the desktop
applications of Linux, however, are beginning to load with “default” applications that need
patching and maintenance. For example, a recent Novell/SuSe 9.2 desktop needed several
critical patches. Most of the “enterprise” ready desktop versions of Linux distributions are
beginning to have patch update tools built into the operating system. For example, both RedHat
and SuSe come with a “windows update” style of automatic update that is useful for the desktop.
The SuSe distribution requests administrator privileges before scanning for patches.
SuSe prompting for administrator rights before patching
Do not, however, “keep password” on the system to maintain separation between the user
account and the Superuser Do (‘sudo’) account. Once the administrator password is applied, the
system begins the scan and reports on the patches needed.
Yast Online Patch window indicating critical patches needed by system
Obviously similar to Windows update, these workstation auto updates work best for standalone
workstations or small networks rather than the large enterprise settings. For those enterprise
environments, the firm truly needs patch tools and scripts to ensure best practices.
The savvy Linux administrator has an additional option. The more advanced Linux administrator
will download the code and patches in source code form and compile them into a form that he will
use to patch the systems before waiting for the vendor to release a patch. This manual approach
should be reviewed to ensure that the procedure does not void any service line agreements
(SLAs). This review takes significant manual effort to keep track of installed components and
what is patched versus what needs patching. As Linux distributions proliferate into the Enterprise
environment, a growing number of firms support patch distribution and inventory tools for Linux.
Page
58
Here are two vendors with Linux patch management tools as a sample:
•
•
Shavlik with their Shavlik HFNetChkPro for Linux 2.1 – for RedHat
Novell with their Linux Management console found here
http://www.novell.com/products/zenworks/linuxmanagement/
When you begin a support agreement or other support arrangement with a commercial Linux
vendor, discuss all the patch management tools they provide with the package itself. Manual
patching and chasing down security patch information is at best tedious for any operating system.
Patch information is available from each vendor. For example, each vendor normally has an
email/newsletter service. Find samples at
•
•
•
•
•
•
•
http://www.suse.com/en/private/newsletter.html
http://lists.suse.com/archive/suse-security/
http://lists.suse.com/archive/suse-security-announce/
https://www.redhat.com/mailman/listinfo/enterprise-watch-list/
http://lists.debian.org/debian-security-announce/
http://www.gentoo.org/main/en/lists.xml
http://www.mandrakesoft.com/security/advisories
Among others, review the offerings from the specific alternative operating system around which
you choose to build.
Here too, as was the case in our Microsoft-specific chapters, standardization on one platform may
go a long way to helping a firm manage “change management.” If your firm had to keep up with
patches for Microsoft, SuSe, RedHat, Debian, Mandrake, OpenBSD, and others, you would either
need to subscribe to a notification service provided by a company such as Secunia
(www.Secunia.com) or devote hours to keeping up to date. Gaining core competency surrounding
a limited number of vendors is central to managing the “patch beast.”
Remember that patching is mandatory for any software application. In late December 2004, the
“Sanity” worm affected web sites that included code from phpBB web forum. Some other
anecdotal issues in the Linux world include:
•
•
•
The “Ramen” worm shutdown print servers.
In 2000, well-known vulnerabilities in the ToolTalk database server compromised Solaris
systems.
“Slapper worm” attacked Apache web servers.
We could go on with more such examples, but the bottom line is each platform has vulnerabilities
and needs patching. As we have stated before, the risk of introducing new code into a stable
system could jeopardize that platform. Destabilizing a system will mean loss of productivity and
revenue, and we need to weigh the risk against the benefit.
Even after applying patches to a system, some firms wait for a “burn in period” or review patch
community results before applying the patches firm wide. Vendors cannot test for all possible
combinations of hardware and software and business practices and thus you must prepare your
own test policies and procedures before implementing patch deployment. As in any introduction
of new code, you may also want to have a “burn in” period to ensure that the patches work
properly in your unique system.
Begin by ensuring that you have secured your systems. The Center for Internet Security
(www.cisecurity.org) has benchmarks for RedHat, Solaris, HP-Unix, and various other systems
and provides auditing tools as well. Bastille (http://www.bastille-linux.org/) is another tool that
Page
59
provides software-hardening information to help you proactively harden your systems against
attacks. Again, it is imperative that system hardening of all operating systems complement patch
management best practices.
Unlike the Microsoft operating systems, Linux system administrators have the option of manually
compiling patches rather than relying just on the vendor. However, this process is not a trivial one
and you may wish to wait for the vendor complied packages due to unique changes with each
distribution. We will discuss this later in this chapter. Ensure that whatever method you choose for
patching is in line with both your operating systems and your applications. Again be aware of your
vendor service level agreement requirements for commercial packages. While Linux distributions
are based on the open source platform, it is your agreements with your line of business
applications that are reliant on the software that you should pay attention to the most. If you have
a commercial application built on a Linux distribution, has your application vendor approved the
patch the question you should be monitoring the closest. You may not have any such
agreements in place to worry about and can patch on your own schedule with your own
methodologies.
As we have previously pointed out, while the patch management process consumes time and
resources to identify vendors, to develop patch testing resources and methodologies, the price of
not patching is very high. Costs of loss of productivity, of reputation, and these days the potential
for legislative requirements to ensure that reasonable security procedures are followed all add to
the “costs” of not patching.
For the RedHat distributions prior to the “Enterprise” versions, RPM (or RedHat Package
Manager) files are the released updates, configurations, and documentation. Packages for Athlon
and Intel processors appear along with a folder called “noarch” which apply to all systems
regardless of the hardware platform.
RedHat 9 RPM locations
For RedHat Enterprise, the packages organize as follows:
• AS (Advanced Server) is the RedHat top of the line package for large enterprise servers
that can support up to 16 CPUs and 64 gb of memory
• ES (Enterprise Server) supports two CPUs and 8gigs of memory.
• WS is the workstation version of RedHat for highly technical needs.
• Desktop supports a single CPU.
RedHat Enterprise packages
Each shop must decide whether to wait for vendor-supported packages or introduce other means
for patching. Also, keep in mind that obtaining the updates by browsing to the site
ftp://updates.redhat.com includes non-security updates. Before you begin the process of
downloading packages from either the RedHat website or any other website, ensure you are
using a trusted site and only use trusted sites. Earlier in 2004, a bogus RedHat security email
attempted to entice people to download a Trojan.
Page
60
Compare the PGP signature (do you cover or point to documentation to support PGP on SBS or
exchange and Outlook?) for emails and md5 checksums of the software you receive.
For example when downloading the Fedora version of RedHat, compare the following checksums
to the files downloaded:
Hash: SHA1
2786a751df919f340d967a4833b63b16
0b9f23eff713db3a0d5a778e38d70299
Bd15bac386cdc30b6fcdf8e7da633403
D89ea5afcb3b94cbfaa3c275455d45a4
D177e8134ffff13e0029fe31a9f20c85
B61b0eb7e0171837aeeff4f0054a4d79
99dc12c7e8a93844a48a5675a9c07ec9
399b7ffd721ebb4244a02c34cdbb1b82
F58b1de3b880df55dbbd37d143419226
FC3-x86_64-DVD.iso
FC3-x86_64-SRPMS-disc1.iso
FC3-x86_64-disc1.iso
Compare the PGP signature to the RedHat public key at
http://www.redhat.com/security/team/key/. To confirm a RedHat package, run the command as
follows: rpm --checksig -v <filename>.rpm
The process of ensuring that checksums match, confirms that you are not in a similar situation as
individuals who downloaded software tools from file sharing locations that contained Trojan
horses in the source code and thereby infected their systems with backdoors. Individuals who
didn’t verify checksums introduced bad code into their systems. Thus follow “good common
sense” when downloading patch sources. You may accept the risk for your less critical
machines, but you may not want to accept the risk for more critical machines.
The RedHat command can be used to determine what version of the package you have on your
system. #>rpm --query –all or rpm –qa can show you a listing of packages installed. By ‘piping’
the ‘grep’ command, (rpm –qa | grep <search string>) it is possible to easily locate particular
applications.
Sample of querying for all software
Page
61
You can also use this to query a specific package such as
#>rpm –q openssh
Verification of Openssh package, please note I am logged in as “root” which should be limited
and only done by administrators.
In general, to update a package you need to follow instructions:85
To update all RPMs for your particular architecture, run:
#>rpm -Fvh [filenames]
where [filenames] is a list of the RPMs you wish to upgrade. Only those
RPMs which are currently installed will be updated. Those RPMs which are not installed but
included in the list will not be updated. Note that you can also use wildcards (*.rpm) if your current
directory contains *only* the desired RPMs.
Please note that this update is also available via RedHat Network. Many people find this an
easier way to apply updates. To use RedHat Network, launch the RedHat Update Agent with the
following command:
#>up2date
this will start an interactive process that will result in the appropriate RPMs being upgraded on
your system.
If up2date fails to connect to Red Hat Network due to SSL Certificate Errors, you need to install a
version of the up2date client with an updated certificate. The latest version of up2date is available
from the Red Hat FTP site and may also be downloaded directly from the RHN website:
https://rhn.redhat.com/help/latest-up2date.pxt
You can verify each package and see who signed it with the following command:
#>rpm --checksig -v filename
If you only wish to verify that each package has not been corrupted or tampered with, examine
only the md5sum with the following command:
#>md5sum filename
To verify software integrity you can use the switch of rpm – verify or rpm –v to check packages
against the local database.
Page
62
Verification of files
To install software you can use rpm – install, rpm – upgrade or rpm – freshen. The ‘install’ flag
ensures the new file is loaded but leaves the old version installed, the ‘upgrade’ flag removes a
previous version that might exist, and the ‘freshen’ flag removes only the prior version of there is
an older version on the system.
A “defense in depth” measure is to not only disable a service but also remove the software
completely, thus if you determine that you do not need a package on your system, rpm – erase is
recommended to remove the software completely.
Kernel updates should only be done with the ‘–I’ flag. To review what Kernel level you have on
your system type in rpm –qa |grep kernel
Verification of Kernel on RedHat Enterprise 9
This will show the kernels installed on the system. You may also consider the #>uname –a
command to determine which kernel is currently loaded into memory and running. Modern Linux
distributions support either GRUB or LILO as a boot manager. This permits a quick and easy
‘rollback’ by just rebooting the machine, and choosing a previously used kernel.
In addition, for those using RedHat/Fedora, up2date is phasing out. The replacement, called
“yum,” evolves from Debian’s update utility. This new tool helps to eliminate some of the issues
surrounding RPM’s poor dependency resolution problems. RPM has occasionally installed
applications rather than libraries to resolve library dependencies. This just added another
application of questionable value, but one that needs maintenance for patches, updates, and
revisions.
Page
63
Another element to keep in mind is that the files downloaded by these tools can be stored in a
shared directory. When other machines mount that directory, up2date need not be run on all of
the machines, just the installation process. This is extremely beneficial to updating multiple
workstations. Usually an administrator will not have several servers to update that all require the
same program updates, this naturally excludes cluster administrators.
For less skilled administrators, we still recommend using a more commercial package such as
RedHat or SuSe that has a scan and update feature. For example, even on RedHat’s Fedora
package, the up2date utility showcases that a recently installed ISO still needs updating:
RedHat Fedora update notification
So taking our example, we first visit the Fedora RedHat web site to get the update at
http://download.fedora.redhat.com/pub/fedora/linux/core/updates/ and download the needed
packages we need so that we can manually update the Omni-0.9.1-7 to the more current version
found on the Fedora web site to Omni-0.9.2-1.1. We would of course first check for
dependencies; when we launch the download we allow the package to check for itself.
RedHat preparing system update
Page
64
We can see from the next screen that the package that it depended upon was not available in the
system and thus it needed another rpm package first before updating the file:
Fedora indicating dependency on another RPM package
Once you have found the dependency, you run the rpm –Fvh command with the series of files
together to begin the patching process
Installation of files on RedHat Fedora
We have now successfully updated these files and can confirm their installation. This process can
be a bit tedious at best. Thus, patch management tools for any platform that you protect can
easily pay for themselves in person-hours, in more automation and confirmation. Patching for any
platform is not easy and takes skill. Using advanced tools to automate this process are highly
recommended.
Page
65
Building packages from source
Because Linux distributions are modular, there is not a “rollback” feature per se, but rather
redistribution, a reapplication of the necessary package to “roll back” to a working version.
Besides waiting for a distribution package from a vendor, you may wish to build from source.
Steve Friedl, a Security consultant, has provided detailed instructions on how to build from source
packages on his web site at http://www.unixwiz.net/techtips/building-source.html. You can review
the steps for more detail.
Another process for distribution in the open source world is called the Gentoo distribution is
gaining popularity in certain circles because the entire OS/distribution is custom built on the
machine. The packages are downloaded and compiled by the installation program. The
disadvantage to this is that it can be extremely time-consuming; however, the benefits include
knowing exactly what has been installed on the machine, no more or less than the files needed
for the system to execute its assigned tasks. This distribution is useful for environments where
peak performance and/or heightened security are needed. In an enterprise environment, (large or
small) Gentoo is best when deployed to the same hardware on or for which it was compiled.
Clusters are a good example, as are machines purchased in ‘groups’ for team environments such
as call centers. Unfortunately, due to the time involved in developing a fully optimized single
machine image, this would not be a recommended solution for environments with many systems
from different vendors, particularly desktops.
As Steve Friedl states in his Technology tip86:
•
•
•
It insures complete control over all the build options, including the installation pathnames.
Using a pre-built package with options you do not need (say, an LDAP client) adds to
bloat.
It insures that cooperating packages are built with compatible options: if Postfix and
Courier-IMAP both need to link with MySQL libraries, building them all yourself
guarantees that the same compiler was used for all of them, with flags you can see.
You develop a reusable skill that applies anywhere you install the package: apt-get is
great for Debian but it won't be too helpful on Solaris or HP-UX.
Dependencies
As with many software patches from many vendors, there are dependencies that must be
followed to install software. You can use rpm–qf, for example, as the anaconda package is not
dependent on any other package. We saw in our own example that our sample patch was
dependent on another package, testing a more innocuous package indicates it does not have a
dependency.
Page
66
Sun Solaris
Sun Solaris recommends the following in their patch management strategy “The Recommended
Patch Cluster reflects the current version of all of the patches required to address all Sun Alert
issues. Some customers have adopted a policy of periodically applying the current
Recommended Patch Cluster to a system. While this strategy does address all Sun Alert issues,
it also introduces more change to the system than is necessary. Similarly, reapplying the current
Recommended Patch Cluster on a scheduled basis is also not necessary. 87
For Sun Solaris, a patch report file is located at
ftp://sunsolve.sun.com/patchroot/reports/public_patch_report. This lists the recommended and
security patches for Solaris as of any given date.
Example of Sun Solaris public patch report
The Sun Alert Pack reflects the lowest revision of all of the patches necessary to address all Sun
Alert issues, and the least possible change to your system while still addressing all Sun Alert
issues. Sun believes that you should apply patches only to address specific issues or needs
Page
67
unless critical. According to Sun, you should not apply patches merely to keep current. There is
no benefit to applying the latest revisions of patches without understanding whether those
patches provide any value.
Patch management tools for Solaris include
• Sun Patch Manager 2.0 http://www.sun.com/software/download/products/40c8c2ad.html
• Solaris Patch Manager 1.0 http://www.sun.com/service/support/sw_only/patchmanager.html
• PatchPro - http://patchpro.sun.com/
• Traffic Light Patchtool - http://www.sun.com/service/preventive/index.html.
• Patch utilities
Coda
We hope that this very brief overview of the methods and options you have when patching nonMicrosoft systems shows you that above all else you need the following key information no matter
what the platform:
•
•
•
•
•
•
•
•
•
•
Information from the vendors for the patches needed for security issues
You need the software patches
You need a test network set up
You need to test the patches against installed applications that you need to ensure
continue to run and perform as expected.
You need to ensure you have a backup of production systems
You need to ensure you have a rollback plan in case the patch does not work even in the
production setting
You need to confirm that the patches have been applied
You need a review process
You need to start the process all over again each time a new patch comes out.
No matter what the platform, if you have “it’ installed, patch “it.”
Page
68
Chapter 6 – Justifying Patch Management—A Business Tutorial
H
e has his laptop. He has printed, bound, and carefully reviewed his customized
PowerPoint and he is finally off to present to one of his biggest prospects! The day is cool;
the weather perfect the feel of a sale is in the air! The last eight months have been
grueling; this prospect is one of the most demanding he has ever worked with and his company
has met the prospect’s every demand.
What our eager sales person doesn’t realize is that his trusty laptop, the one he carries with him
every where, the one that has been to the beach, and the one that has survived being dropped at
the local Starbucks is about to cost him this deal.
Management and Patch Management
Why should any sales director be concerned with the company’s current information technology
infrastructure? The sales director is responsible for a team of people who, in general, are not
going to be cutting edge IT specialists. They need access to prospect information, the ability to
write a letter or an e-mail, and internet access for research, but not necessarily much more. The
sales director is responsible for insuring that the team is successful, efficient, motivated, and
passionate about their position. Would a sales director be interested in reading a complete guide
to patch management? Highly unlikely, but why does management need to be cognizant of patch
management best practices?
Why? A sales person visits a prospect. The sales person walks in with a portable computer with
an innovative presentation that the marketing department just prepared. He sets his computer on
the table and connects with the prospect’s IT person. He turns the laptop on and plugs it into a
projector. During boot up, the IT person at the prospect’s site notices that the machine boots to a
Microsoft Windows 98 screen. What is the first impression? This prospective vendor and his
company do not know how to protect themselves from vulnerabilities and risk, “that the company
is using old software,” “that the company is using an unsupported operating system!”
The IT person can only feel relief that the sales person has not asked to use the internal internet
connection. “We could have been infected in less time than he plans to be presenting (at best 20
minutes).” Now consider the last presentation that you attended. Did you notice the equipment
used? Have you personally questioned the technical competence of presenters? Did you know
that even in security-focused presentations the audience could be shocked when the presenter’s
laptop has the icon in the toolbar indicating that a security patch’s status as downloaded but not
installed?
These two situations indicate that the decision of each person’s company “not to keep current”
just cost the sales person a sale and reduced the reputation of the presenter and the company he
represents. What is worse is that they do not even know what hit them! Before the presentation
even started, the prospect’s IT person is projecting a negative attitude and the technical people in
the audience are prejudiced even before the sales person opens his mouth, and the nontechnical attendees are picking up bad vibes.
I can sympathize, as I have been in this position. Buyers judge salespeople on their looks, their
knowledge, their articulation, and the tools that they carry. Keeping a sales representative’s
technology current is a huge concern for the sales director. Our sample scenario involves not only
a lost sale, but also seriously compromised sales staff motivation. They feel abandoned when
they factors outside their control nail them. Getting a meeting can be an extremely difficult and
can occasionally take years. It is critical to create the best first impressions in first meetings, as
most sales managers understand. Keeping a company’s machines current and secure sends a
message to the buying audience that your company cares about security! Compare it to the
Page
69
“infection” impression you might get if someone wiped his nose before offering to shake your
hand!
Now apply our sample insecure scenario to a larger sales event such as a trade show. Most trade
show exhibits rely on shipped or rented computer technology. This equipment runs slide shows,
highlights product benefits, demonstrates product features, and collects prospect and client
information. When did you last patch your company’s trade show equipment? Consider that larger
companies often dedicate some equipment for use at shows only. Among some of the larger
vendors, you would assume that the trade show equipment was well “oiled and greased” before
shipping.
However, a malicious worm called Slammer hit during the American Institutes of Certified Public
Accounts’ 2004 annual technology conference. The vendors all had invested thousands of dollars
in booth space for the audience of almost one thousand technically inclined CPAs. The show
included an audience of firms working with almost every business in the United States and was
an attractive venue for reaching businesses. For a vendor a trade show also includes additional
costs including equipment rental, the care and purchase of the booth itself and staff expenses;
and yet over half of all the vendors in attendance had no way to conduct business when Slammer
hit. This included some of the largest computer vendors in the world and some of the more
technically advanced vendors in the country. Vendor representatives were depending on
attendees for help and vendor booths were “dark.”
The impression was significant! Management was oblivious to patch management best practices.
Some are still recovering. We can agree, then, that patch management best practices affect
image, marketing, and sales, but they also affect other departments, as well.
An Officer’s perspective
We can view corporate officers’ responsibilities from several angles. There are the big picture
considerations and decisions that require a forward thinking individual who constantly looks
toward what ultimately should happen. Then there are getting the tasks done to reach smaller
and then progressively larger goals. We refer to the above as the doers and the thinkers. The
thinkers can artfully decide where a company needs to be in a month, a year, or in five years to
grow and prosper. The doers have a way of using the available resources in a company,
including the specialized skills of team members, to achieve the smaller goals and to move
toward bigger picture items. Patch management or the lack of patch management procedures
affect both of these types of leaders and their daily environments.
The CEO, President, and Founder are generally big picture thinkers when it comes to a business.
The CEO must constantly be aware of the environment in which a company does business. He is
the captain of the ship and needs to know about the possible iceberg formation hundreds of miles
away, of parallel business traffic that might interfere with the company’s plans, and of
environmental variables. When it comes to security patch management, the CEO needs to know
of the level of risk associated within any given period. He also must be the compliance manager!
Vulnerabilities and risks, created by the very nature of technology, affect big picture decisions for
moving forward and must be on the CEO’s agenda. If we consider that the global environment in
which we all do business is subject to government regulations, a war, or other factors that can
increase the risk, then the CEO must be extremely sensitive to any “exposed” sides of a
company.
Additionally today, the CEO must also be concerned with his own personal risk due to noncompliance. Given that the technology infrastructure of a company is in constant change, the
exposure to risk is a changing variable that needs constant attention. To the CEO, patch
management is about best practices and compliance; it is about reducing risk and vulnerabilities;
and it is about protecting the corporation, the corporate officers, and the team members. Consider
that Gartner predicts, “By 2005, 60 percent of security breach incident costs incurred by
Page
70
businesses will be financially or politically motivated.”
increases corporate risk.
88
Weak or ineffective patch management
The Chief Information Officer or Chief Technology Officer focuses on the use of technology within
a company. This position emerged in the United States in the 1980s as a business-focused
extension of the position of Director of R&D. Large research-oriented companies like General
Electric, AT&T, and ALCOA created this position to increase the profits derived from research
projects in their laboratories. During the dot-com and computer boom of the 1990s, many
companies used the CTO title for their senior technical person. The MIS and IT community often
use the title CTO as either synonymous with Chief Information Officer, or as a subordinate to the
CIO who is more versed in the technical intricacies of the systems being deployed. There is no
uniform application of the title and this causes confusion when people across domains discuss
the role of this person. The roles of the CTO vary among companies and within industries, but
89
usually relate to technology. Immediately we realize that creating and rolling out a patch
management best practice environment is closest in responsibility to the role of the CTO, but do
patch management best practices help the actual IT departments? Does a patch management
best practice make a CTO’s job easier on a day-to-day basis?
Information Technology departments are traditionally environments of crisis management.
Depending on the day, a set of workstations could be down, a database could be corrupt, or a
network could be having trouble that requires the attention of the IT department staff. The
Director of IT must also deal with limited resources to manage these crises given the erratic
nature of the demand for IT professionals within a business environment. When a new application
rolls out to a department, the demand for support significantly increases. On the other hand, if the
application and infrastructure environment has not changed over the course of a few months the
demand for IT support depends on outside variables that affect the internal infrastructure.
In no other department can patch management best practices or a proven patch management
process have a bigger impact then to reduce the demand on the IT staff. Small changes to
applications and operating systems create efficient environments for problem resolution. If the
corporate policy is to wait to install updates and patches, than the changes become bigger and
bigger depending on how long the wait extends. Not only do the changes become bigger by
accumulation, but the vulnerabilities increase as more and more programs that exploit
vulnerabilities appear. We must also consider that the longer a company waits to role out a patch,
the further away the vendor who created that patch gets from its development. This creates a
situation where it is harder for the original vendor to make additional changes based on the
idiosyncrasies of any one customer.
Specific industries also must contend with special issues. In the medical community, for instance,
the CTO balances the needs of patch management as recommended by the security provisions
of HIPAA rules with the restrictions that medical equipment vendors impose to ensure that their
equipment has a controlled change process in place. We will focus more on industry specific
niche issues in further chapters.
This brings us to the Chief Financial Officer (CFO), the accounting manager, or controller. “A
financial manager, through his understanding of the company's financial health, the current
90
market, and the goals of the company, helps set direction and guides decision making.” The
financial manager must understand more than numbers. He must be able to interpret data, be a
researcher, and comprehend what needs to occur within his sphere of influence to help a
business prosper. This includes balancing business needs/decisions/goals with investor’s
needs/decisions/goals. The financial outlay and the investment that is required for rolling out any
patch management best practice solution usually becomes the first consideration for the CFO;
however, CFO’s are also very concerned with risk.
Page
71
How do you measure the impact of patching versus not patching against the investment needed
in both time and dollars to roll out a proper system? There are certainly the tangible and
intangible variables and we will consider these as we look at ROI, but remember that downtime
costs money, sometimes so much soft money that measurement becomes extremely difficult!
Downtime can also create either more of/or less of an impact depending on the nature of the
business and the industry niche. Technology is also its own industry niche and technology
variables affect patch management best. The CFO who compiles and creates the ROI around
patch management must be more than a CFO. He must be a CFO who understands technology
and the balance of risk within the technological world.
We then have the Board of Directors, a group of executives often not located with or employed by
the corporation. Most managers however do report to a board, but when it comes to patch
management best practices a board of directors has a very different focus. They are independent
of the firm but how they manage themselves, and how patch management best practices impact
the board can cost a firm significant funds and loss of resources. Consider the cost of the board
of directors to a corporation. Senior executives are expensive! Now consider what they are
offering. Board members offer insight, experience, connections, and opportunity. They offer
oversight and management support.
The board has experience with the industry as a whole, with acquisitions and mergers and with
the potential opportunities to a firm. With a board ranging from six to nine members, this
bandwidth should be making powerful decisions and often they do—but—and the big “but” comes
in the flavor of access to information and managing that access. How can the board make
decisions without the appropriate information or without accurate access to current data? Printed
reports a few months out of date are not easily digested, nor very applicable in an extremely fast
changing environment.
One of the epidemics in the corporate culture is the lack of available access to or appropriate
control of digital information within a corporation. Board members are often corporate members of
other companies, their internal privileges need monitoring, and their access is sensitive to
changes. Yet they are often forgotten when changes need to be made. We find poorly rolled out
patch management best practices can easily compromise the sensitive relationships that exist
between a board and the management team, and repairing these relationships is not an easy
task. If the BOD has no access to appropriate and current information then the company is not
getting full benefit of their expertise. "There is a delicate balance between limiting insider access
to information and crippling the ability to create revenue," said Richard Hunter, vice president for
Gartner. "Generally, this conflict between security and commerce is resolved in favor of creating
91
revenue and therefore facilitating insider crime." Gartner advises that among other actions,
businesses must create and enforce legal agreements defining legitimate use of proprietary
92
intellectual property by trading partners and employees. These legal agreements can also fall
under the security arena best practice umbrella. Now consider, does the CTO always have deep
insight into the existence of these legal agreements?
When designing patch management best practices carefully consider every department within a
company and every person who needs access to all data. Plan how to keep data available to
those who need access and how to comply with existing legal agreements—not only how but
which legal agreements apply to best practice policies.
Control board member access by network and files rights management techniques such as
limiting data and file access to read only or storing corporate data that is unrelated to the duties of
the board in folders that are inaccessible to the BOD. To use the power and connections of a
board effectively, a company must train and work with these senior level executives to ensure
they can use the power of data mining and business intelligent tools.
Page
72
We turn our attention now to sales, marketing, and customer service. What about the every day
tasks that employees and staff must accomplish to meet their job requirements? As more and
more companies depend on technology to meet the needs of prospects and customers, problems
can affect every department’s productivity.
Marketing, consciously or unconsciously, integrates with almost every process within a
corporation. Most corporations have a marketing department focused on the task of marketing
externally by polishing a company’s image by preparing financial statements, producing and
promoting products, or delivering compelling customer service. Marketing is part of every step of
the business process.
Technology is close to same role. When we call a client, we look up a telephone number (using
technology). When we bill for a product or service we use accounting information systems
(technology). When we market or advertise a product, we design mailings, make calls, or put
advertisement in the news (using technology). We talked about the impact of poor patching on
the sales process and certainly, our example plays to the perception for the new prospect. The
firms impacted by the SQL Slammer and Blaster viruses realized quickly that not only did they
have a “technology” issue, but a “marketing” issue. The fact that their firms’ names appeared on
the nightly news as targets of these events caused issues that their marketing or public relations
departments had to deal with.
What other issues can an unpatched or even up to date systems cause? If a machine within the
marketing department is not current, updated, or patched, it can become an issue with the
outside vendors who work with that department. For instance, a publisher might not accept an
electronic file needed for advertisement, because it is not in a compliant or specific format. This
issue limits the marketing department’s efforts to create brochures; company t-shirts, mass
mailings, or other products in compliance with outside requirements. In smaller firms, having the
appropriate version of the software to produce the correct files can be a daunting task, but it is a
rare magazine vendor who will accept a digital advertisement that is not compliant with the latest
digital format. Having technicians to keep a firm current and up to standard increases the internal
efficiency of file sharing as well as the external needs of file sharing with other vendors. When a
company’s tools are current, that company is in a better position to compete.
Security issues within marketing create layers of policy that affect standard practices such as emailing files. In late 2004, a security issue with image files caused some firms to begin to analyze
whether or not to block such attachments. Industry wide best practices to block image file
attachments slammed the marketing department’s use of e-mail to share files with publishers.
A Patch Management Best practice includes knowing what types of files are in transport at any
given time within a firm or department. Using policy to single out what is permissible within the
marketing department increases the department’s ability to work while allowing IT to focus on the
most relevant threat vectors. Blocking all attachments cripples the marketing department and
crisis management would kick in to resolve the issue.
Keeping systems on standards platforms and consistent with Patch Management Best Practices
assists in ensuring that a business stays agile, out of crisis management and proactive in it’s
ability to ensure interoperability with other business partners.
Customer service is an ideal place to gather customer information and many firms fail to take
advantage of this opportunity. Without patched, functioning, and current CRM software, data is
continually lost, even though data accumulated equals powerful information for a firm. A
dysfunctional CRM system is worse than having no CRM system because when customer service
representatives run into a problem with the system they are usually on the phone with a client or
prospect. Nine times out of ten the client or prospect has the impression that the corporation
cannot keep its computer systems running. Talk about direct ding!
Page
73
Lastly, we must consider that at the heart of every business is the supply chain and cash flow!
The supply chain (inventory control, manufacturing, purchasing, etc.) is truly the blood that keeps
a business alive. Consider that any company that manufactures a product relies on machinery to
produce that product and, more and more, machinery relies on software to function. We must be
keenly aware of what sort of problem this type of software dependency would cause. Machinery
is becoming comparable to a living, breathing, and changing creature that needs careful
management and maintenance, and the biggest management component of machinery software
is patch management.
Much of our world relies on software and this phenomenon is only going to grow. Consider the
number of electronic devices used in any given environment; you have to accept that without the
proper functioning of these critical devices we would be in serious trouble. “The epidemic of
Windows-based worms and viruses in the past year has put hospital IT administrators on a state
of high alert to protect patient-care systems that have become reliant on Microsoft operating
93
systems.”
It is not just the Microsoft operating system at risk. More and more sophisticated hackers are
wasting time creating havoc. The end of 2004 saw a barrage of Oracle database vulnerabilities
released to the web right before the Christmas holidays. The Oracle Database infrastructure is
not dependent on the Microsoft operating system. It runs on a variety of platforms with a variety of
operating systems. For all firms using Oracle databases and software, crisis patching
requirements increased at a time when business was busier than usual.
How can businesses deal with the risks generated by external security threats and documented
by researchers who discover and document these threats? How can the best practices help us
reduce risks? Understand your unique environment, build strong relationships with vendors, and
implement business-to-business (B2B) best practices that include high quality, partner-level
relationships. Such relationships provide you with options, agility, and customized resolutions.
This book focuses on many different elements of patch management best practices, the most
fundamental of which is building and maintaining relationships at levels that create good flow of
information, partner-to-partner win/win resolutions, and communication.
To understand fully each manager’s risk assessment role, review the following guidelines
provided by the National Institutes of Standards and Technology.
Senior management, under the standard of due care and ultimate responsibility for mission
accomplishment, must provide the necessary resources to develop the capabilities needed to
accomplish the mission. They must also assess and incorporate results of the risk assessment
activity into the decision making process. An effective risk management program that assesses
and mitigates IT-related mission risks requires the support and involvement of senior
94
management. Senior managers have a level of power that can cut through layers of people
and “peon filtering.” Senior managers need to build bridges between the appropriate people at
vendors the firm depends on, allowing staff to connect with the appropriate people at other
corporations.
Chief Information Officer (CIO) is responsible for an enterprise’s IT planning, budgeting, and
performance including its information security components. Decisions made in these areas
95
should evolve from an effective risk management program. The CIO attends trade shows and
industry events, and mixes and mingles with other CIOs in the specific niche. The CIO also
communicates information regarding threats to a given industry.
System and information owners are responsible for ensuring that proper controls are in place
to address integrity, confidentiality, and availability of the IT systems and data they own.
Typically, the system and information owners are responsible for changes to their IT systems.
Thus, they usually have to approve and sign off on changes to their IT systems (e.g., system
Page
74
enhancement, major changes to the software and hardware). The system and information owners
must therefore understand their role in the risk management process and fully support this
96
process.” System and Information owners are classic roles for participating in forums known to
discuss the current risks. These forums include patch mananagement.org as well as industry
specific niches focused on security risk analysis. Relationships with technical experts in the
security space increase the resources a system and information owner brings to a corporation.
Business and functional managers responsible for business operations and IT procurement
processes must take active roles in the risk management process. These managers have the
authority and responsibility for making trade-off decisions essential to mission success. Their
involvement in the risk management process enables the achievement of proper security for the
IT systems, which, if managed properly, will provide mission effectiveness with a minimal
97
expenditure of resources. The business and functional managers bring an outside perspective
to the role of creating patch management best practices. Using their relationships, they can
communicate options and case studies about how others within the corporate world are reducing
risks, giving enough variable and perspective to pick the most efficient path.
Information system security officers (ISSOs) and computer security officers are responsible
for their organizations’ security programs, including risk management. Therefore, they play
leading roles in introducing appropriate, structured methodologies to help identify, evaluate, and
minimize risks to the IT systems that support their organizations’ missions. ISSOs also act as
major consultants in support of senior management to ensure that this activity takes place on an
98
ongoing basis. Given that change happens so fast in the world of technology and even faster in
the world of security, the best technologists and have depth of contacts unmatched by their
peers.
IT security practitioners, that is, network, system, application, and database administrators;
computer specialists; security analysts; security consultants are responsible for implementing
security requirements in their IT systems. As changes occur in the existing IT system
environment (e.g., expansion in network connectivity, changes to the existing infrastructure and
organizational policies, introduction of new technologies), the IT security practitioners must
support or use the risk management process to identify and assess new potential risks and
99
implement new security controls as needed to safeguard their IT systems.
Security awareness trainers (security/subject matter professionals) school an organization’s
personnel in the use of the IT systems. Use of the IT systems and data according to an
organization’s policies, guidelines, and rules of behavior is critical to mitigating risk and protecting
the organization’s IT resources. To minimize risk to the IT systems, it is essential that system and
application users have access to security awareness training. Therefore, the IT security trainers
or security/subject matter professionals must understand the risk management process so that
they can develop appropriate training materials and incorporate risk assessment into training
100
programs to educate the end users.
Industry Requirements/Standards
Healthcare Industry—in no other niche environment can this be better illustrated then in the
world of medicine. A healthcare facility provides a heart cathode system that includes a couple of
personal health Information (PHI) servers attached to the facility’s network. In security scans of
the servers, the head of the IT team has noted that the manufacturer is not maintaining operating
system patches and will not approve running antivirus software on these devices. The local IT
team has been diligently working with the manufacturer to have the holes patched. It is not that
easy, as the manufacturing vendor has made no progress. Meanwhile, the epidemic of Windowsbased worms and viruses in the past year has put hospital IT administrators on a state of high
alert to protect patient-care systems.
Page
75
The hospital IT team has been diligent and has installed a monitoring software package on the
servers to allow remote access to the systems. This software identifies a rogue dial-up modem
and a few other violations of the normal system setups; however there are additional problems as
the systems came with a virus scanner that the vendor updates and the vendor is updating this
virus software only once a month. This standard for virus detection is outside the local IT
department’s best practice policies. With the local team, the best practice standard is to install the
virus definitions as they became available from the anti-virus vendor.
It gets even more complicated as the manufacturing vendor has asked the local hospital IT team
to remove the monitoring software and the anti-virus scanning software, stating that they cannot
101
support the system running software from other vendors, as doing so skirts FDA approval.
Often, networks allowed to grow organically are not properly isolated and, as such, lie outside
proper protection. All businesses need protection, but rolling out a patch does carry risks because
rolling out a new patch is introducing new code and managing these risks is the goal.
The manufacturing vendor is ensuring their due diligence about testing and certifying patches and
the world at large needs to be diligent about expecting and requiring vendors to raise the security
bar by approving patches in a reasonable period. Some vendors within the medical equipment
niche have pointed to United States Federal Drug Administration rules mandating “pre-market
submission tests” as the reason for their delay in approving patches. While this might make
sense, it is not good enough, particularly given the online arguments against such statements.
“The FDA says fingers shouldn't be pointed in its direction,” argues one poster. “The medical
device manufacturers say they need FDA approval, but they don't," says another. "If this is a
102
security or virus-related patch, it doesn't require the pre-market submission,” notes a third.
“Today, the delivery of healthcare increasingly relies on medical information systems (MedIS).
Such systems rely on modern information technology to digitally collect, process, distribute,
display, and store patient data. MedIS, like other IT-systems, are vulnerable to malware attacks.
MedIS owners and operators have a special responsibility to shield their systems from malicious
103
attacks,”
and patch management is a critical component of this responsibility.
If we look at another example we can consider the process of medical images.” Medical scanners
hit by viruses and worms can be put out of action for weeks while manufacturers decide how the
problem can be fixed and whether patches to prevent future problems can be applied,” an
104
investigation by E-Health Insider has found.
“The fact that many medical scanners now run on
Windows operating systems connected to hospital computer networks to enable images to be
stored and shared compounds the problem. This means that, unless patches are promptly
applied, a scanner running Windows and connected to such a network is susceptible to viruses
and worms designed to attack Microsoft software.” This system could then be down for weeks.
Despite such potential devastation, “not all manufacturers have a clear policy about how and
when new security patches should be applied to their medical imaging devices, and in some
cases will not allow hospitals to apply security patches themselves.”
The dilemma in healthcare becomes the quagmire of who is responsible for what compliance.
The scenario involves “healthcare IT professionals [who want] to patch networked medical
devices and run anti-virus software against them, but say their hands are tied by medical device
makers and the FDA,” and the “medical device makers [who are] balancing the need to keep their
products from getting infected by viruses and worms with making sure the devices meet FDA
requirements” and the “FDA [which] requires medical device makers to adhere to strict
105
manufacturing processes, but wants IT pros and device makers to work out their differences.”
The telecommunication industry differs from the health care industry in that it tends to deploy
patches when they become available into a heavily tested inactive environment. There is no
known standard, but generally rollout occurs on an as needed and when available basis. Vendors
handle patches as part of the maintenance contract. Because systems carry live voice traffic, the
Page
76
telecoms have to schedule the updates for non-peak times, which are generally around 2am.
They bring up the inactive side of the equipment, run it through its tests, and then switch over to
the active side when they are ready.
Once network admins deploy a patch to a live network they test on the live network making test
calls of all sorts inbound, outbound, tandem (in to out), announcement, call failure, and more.
Even with these industry standards, there are historical instances where software upgrades did
not go as smoothly as they should have.
For instance, in 1998 AT&T experienced “a unique sequence of software flaws that caused an
106
outage”
“AT&T said the problem began when a computer command to upgrade software code
in one of the network switch's circuit cards created a faulty communications path that generated a
large volume of administrative messages to other network switches. As a result, the other
switches quickly became overloaded and stopped routing data from customer applications for
periods ranging from 6 to 26 hours before the network was fully restored”
The resulting impact caused disruption for AT&T’s networks and their frame relay customers
resulting in a frame relay network that was out for 26 hours—yet it could have been much longer.
“Praising Cisco for its cooperation in identifying the root cause of the problem and designing a fix,
AT&T expressed “confidence in Cisco and its products.” AT&T will share its analysis of the
network outage with the FCC, the industry-wide Network Reliability Council, and other network
providers. “By sharing this information and best practices,” and AT&T spokesperson said, “we will
help customers avoid similar network outages no matter which carrier provides their frame-relay
service."
A 2002 Agilent white paper documented the scope of the event. Disruption to customers lasted
up to 26 hours and included the failure of mission-critical bank transactions. The fact that Service
Level Agreements (SLAs) had promised 99.99% availability and four-hour recovery times (meant
that) approximately 6600 customers were not charged, resulting in millions of dollars in lost
107
revenue alone.”
Each equipment vendor to telecommunication providers has a different procedure for applying
patches, and there may not be a documented procedure within a particular company. Deploying
patched is just part of their maintenance procedures when they are working on the equipment. On
the other hand, technicians generally deploy updates throughout the entire organization because
vulnerability may affect call processing, and if there is an issue, everyone is aware of it, including
customers. The vendors do as well, as the CLEC pulls down and applies the updates from the
Website. It just depends on the type of equipment. The equipment may be the actual voice
switching equipment, fiber optics transmission, digital loop carrier equipment, channel banks, or
DSL equipment.
Many of the costs to Service Providers caused by outages and downtime are tangible and easy to
measure, such as
• Direct costs of service restoration and equipment upgrades
• Loss of revenue during outages
• Penalties associated with customer SLAs
However, the real costs include some losses that are harder to quantify but may be far greater.
For example,
• Lost revenue from dissatisfied customers moving to competitors or taking new
business to competitors
• The cost of a tarnished image; the lessened ability to credibly market future premium
differentiated services
Wall Street, Chicago Board of Exchange, and the world of finance is another world in which
patch management and risk management best practices are of concern. In the world of fast
decisions and even faster trading, Wall Street assesses downtime in the millions of dollars. There
Page
77
is no room for vulnerabilities that can slow the trading process, for a slow down can mean millions
of dollars in cash flow. The question in the world of finance is not when to update, but to maintain
reports proving the system is patched and current. That the patches were tested and that best
practices are in place. The financial world delegates risks to trading, while reducing the risk of
downtime to the smallest possible variable.
"Our organizational culture has been developed around an offensive mind-set, with innovation as
a critical underlying theme,’ says Guy Chiarello, CIO of Morgan Stanley's Institutional Securities
108
Group.”
In the financial industry, firms use their technology expertise to competitive advantage
and as distinguisher factor. “At Morgan Stanley, one aspect of technology innovation is providing
a better understanding of risks. [we have} a position, called IT risk manager that pulls together all
the issues that underlie risk for a technology organization. This includes document and instant
messaging retention, data security, market and credit risk, operating risk, and compliance with
109
Sarbanes Oxley and Basel II.
"Whether it is data security, security around client technology
offerings, or overall operating risk, risk management—and all it encompasses—now requires
110
greater focus," says Chiarello. “
There are some high profile examples of where patching and poor patch management process
caused trouble. Consider, for instance, PayPal’s October 2004 incident involving a poorly tested
update. “PayPal started experiencing problems with its service following a monthly software
update. Officials at the company, owned by eBay Inc., blamed a software glitch for intermittent
111
outages that cut across account access as well as payment, shipping and other services.
Not
only did PayPal customers suffer from lack of access and a screeching halt to their business, but
also PayPal had to recover from the public relations hit.
Remember we mentioned the marketing aspect, and the impact on customer service – you can
bet that the PayPal customer service team was working triple time this weekend. We also talked
about sales, how would the above affect your sales team? Do you think “pending sales” that were
about to close on Monday would close? We can see the impact of one incident is no by reviewing
what came out in the news for the next few weeks about the problems with PayPal and the
management of updates.
What are the ties between patch management and disaster recovery? Disaster recovery is the
umbrella under which patch management resides, but there is more. Mishandled patches can
also cause disasters that a corporation may be ill prepared to handle efficiently. A CTO at a rather
large enterprise recently revealed that backing up an Oracle Financial system and database prior
to applying a patch took 10 hours. Applying and testing the patch for proper installation took
another 5 hours and, if a problem developed, the restore took another 12 hours. Now consider
that we just discussed at least 27 hours of potentially billable time that needs accounting for by
the people who work with the Oracle Financial system. The financial concerns are huge! The time
concerns are huge and the need for storing patches for recovery is critical.
The software vendor perspective views our global environment and is aware of the options and
risks associated with this environment. Patch management ties into security, disaster recovery,
and IT best practices, but it also ties into a deeper understanding of how software vendors create
software, how they create and release updates, and how to work most efficiently within a B2B
model. The corporate relationship between software vendor, reseller, and business consumer is
critical to businesses that depend on technology.
We all know that software has bugs and will continue to have bugs. There is no known quality
control process to eliminate all vulnerabilities in software because users and other applications
significantly affect any specific software application. If we can’t eliminate all the vulnerabilities
within the software and the people who use the software, then what better relationship to nurture
than that with the team who knows the software code better than anyone else in the world! In
response, the vendor who produces and releases software is generally the number one firm with
Page
78
which to nurture a relationship. For the bigger software vendors we must develop bridges through
the reseller or user groups to insure so that we understand vulnerabilities and stay ahead of the
patch management process.
Cyber crime persists. In the U.S., during the five years from 1999 to 2003, attacks on computer
servers increased by over 530% to 137,000 incidents. This is partly attributable to vulnerabilities
in software code, which have grown from 500 in 1995 to over 9000 in 2002. Developing countries
are also susceptible, even after relying on leapfrog technology. Brazil has seen hacker attacks
increase by at least 100% yearly since 2002.These growing numbers bear particularly important
on the financial sector. The International Data Corporation reported that more than 57% of all
hack attacks last year targeted the financial sector. The FBI has corroborated this statistic.
Equally troubling, FINCEN’s Suspicious Activity Reports for Computer Intrusions have shot up
more than 500% over the past year. With the growing amount of financial data stored and
transmitted online, the ease of computer intrusions adds to the severity of traditional crimes such
as identity theft. To put this in perspective for the digital age, over $222 billion in losses accrued
112
to the global economy because of identity theft.
We can all agree that cyber crime is not going
away and as more and more countries and people use computer systems, the law of numbers
comes into play. We can assume that vulnerabilities will increase, that software will continue to
change and that remain patched is one of many of the defenses we need to use to reduce risk.
Best practices continue to spread across the growing global economy. Government regulations
continue to assure that the mistakes made by a few do not compromise those within the greater
expanse of the world. These mistakes can include pure malicious intent such as that which
occurred at many of the large corporations famous in the news and no longer in existence today,
to deficiencies in services and products from pure incompetence. Best practice methodologies
cross numerous industries, though some are niche specific. They include methodologies such as
Six Sigma and compliances such as ISO900x and government regulations that need to cross all
industries like the Sarbanes Oxley Act of 2002. With software controlling much of the world we
live in, the best practices around software maintenance mix with the methodologies in place to
insure high efficiency, high profit, and low risk. Consider how Sarbanes Oxley affects a patch
management best practice.
Sarbanes Oxley is a complex document, but we must consider it when thinking about the
network and computer systems that handle any financial data. Detailed auditing and compliance,
computer glitches, hackers, and poor security are no longer acceptable errors in the business
world. Accountability and compliance rule the day. If we consider the following four interpreted
sections of SOX, we can quickly see that the control of internal systems is very much the
responsibility of the management team for each corporation. Yes, management can delegate
responsibilities, but they cannot forget them.
The AICPA Business & Industry Team assembled an ad-hoc task force of members to address
issues relating to management's responsibility with respect to Section 404 of the Sarbanes-Oxley
Act of 2002 titled "Management Assessment of Internal Controls.” “The ad-hoc task force
assembled a list of key issues to communicate to their peers in other companies that are similarly
113
impacted.
The following is a list of four of those key issues and examples with regards to the
risks associated with network systems and patch management best practices
“Management is required to document the system of internal control over financial reporting. As
required by the Sarbanes-Oxley Act of 2002 (SOX), section 404 (Management Assessment of
Internal Controls), management will be required to assess the effectiveness of these controls.
The ASB believes that the evidence management uses to support its assertion about the
effectiveness of its internal control also should be documented. The ASB believes that a failure to
document the system of controls or the evidence used in making the assessment should be
considered a weakness in internal control.”
Page
79
Documentation of financial data is more than a report, more than extracted data and/or compiled
data and certainly contains numerous levels of detail. Most of the time base data comes from
repositories on computer systems. If management is required to document the system for internal
control then they must consider and then incorporate how to validate and protect the financial
data on the network. How should managers control this data? How, for instance, can an admin
confirm that a physical error on a hard drive has not caused a problem with the data residing in
that location? How can he ensure that redundant or malicious data has not been inserted?
Certainly reconcilable reports are one answer, but not all companies take this simple step. As we
move away from paper, we must continue to consider internal controls and the teams responsible
for these internal controls.
“Positive testing of internal control must be performed to make the assessment under SOX
section 404; inquiry alone is not adequate testing. Negative evidence, such as proper financial
reporting, is not evidence of good internal control. On the other hand, a material misstatement
detected by the auditor’s procedures that was not identified by management ordinarily is
indicative of the existence of a material weakness in internal control.”
Several consulting engagements have discovered data being reconciled actually crossreferenced itself. How can this be? What does this mean in real terms? Let us say you are using
accounting system XYZ Software. It handles all of the financial data and accounting for your
company. If you decide to run your controls using a spreadsheet software package then there is
no conflict as long as the source data you are retrieving is independent of the XYZ Software
product. If you are manually entering the amounts into the spreadsheet package from a report run
out of the accounting system then the two are simply duplicates and reconciliation would be
ridiculous. How can this happen? Quite easily, when you consider the superficial knowledge that
most users have when it comes to computer systems. Controls remain valid if practitioners
retrieve the data compared independently and enter it using two different systems. How does
patch management come into play when talking about accounting software? The accounting
software itself needs regular patching.
“The documentation should encompass all significant account balances and disclosures. The
company’s auditors may be utilized to help in the documentation of controls; however,
management cannot abdicate its responsibility for the documentation.”
How many people do you know understand the true nature of writing custom reports and are
proficient in using data mining tools? What would happen if a malefactor compromised a system
prior to extracting data? Do you think that the data retrieved would be accurate? If a user chose
to extract only the summarized A/R data from one field within an accounting system (the
summarized A/R for the month) and a transaction improperly updates the A/R transaction data
but not the summary within the database will the totals for the extracted A/R data be correct?
How would the user know? What if the detailed A/R was purged? Many of the controlling
techniques are standards that go back to the days before computer systems, yet proper
management of data has a significant impact on the ability to reconcile. The handling of data
when posting, updating, and/or simply using a computer system is at the mercy of the
vulnerabilities of the software itself. Since an unpatched system is at higher risk, data posted to
multiple locations can be inaccurate.
Consider the core operating system software or the development software used to write
applications. The OS and development tools are also software packages that can have
vulnerabilities. In fact, when programming during the Y2K crisis, we discovered that the core
numeric fields within the software language we were using was truncating numbers to two
decimal places for display purposes. This truncation was not rounding; it was simply dropping the
last numbers. The displays would print and, as such, multiple displays when compared to reports
were not reconciling. If as little as a penny of discrepancy can mean millions of dollars worth of
differences (which is often the case in accounting) how do you think this error within the basic tool
Page
80
set affected the job of the accounting manager? Numerous technical support calls and hours of
research were required to report the bug, retrieve the patch, and update systems. What would
happen or what does happen to the firms who are not reconciling as closely when errors appear
months later? Who is impacted? The accountability is with the management team. The
management team must be diligent about best practices! Policies should specify addressing
vulnerabilities as they appear.
When using outside service organizations (e.g., data processing, payroll processing, etc),
management should consider the activities of the service organization when making an assertion
about the effectiveness of the company’s internal control over financial reporting. A service
organization typically engages an auditor to examine and issue a report on certain controls at the
service organization, and this report has historically been a tool used by the independent auditor
in planning the nature, timing, and extent of the auditor’s procedures. If a service auditor’s report
on controls placed in operation and tests of operating effectiveness is available, management
may consider whether this report provides sufficient evidence to support its assertion. Since the
date of the service auditor’s report is not likely to correspond with the entity’s end of fiscal year,
management will need to consider what changes, if any occurred at the service organization
subsequent to the service auditor’s report and the effect of the time lag on their assessment.
When significant changes occur, or a significant time lag exists, management may need to apply
their own procedures at the service organization with respect to those subsequent changes or
significant time lag.
As illustrated above a company’s patch management policy can affect every department. If you
do not have best practices for patch management in place, you cannot expect your partners to
have them.
Compliance with ISO900x
ISO900x covers a wide spectrum, but we will focus on its focus on quality control and quality
assurance. What is amazing is that it is not that different from a big picture view from the various
other standards we have been discussing except that it has a long history of tagging a company
as a quality player internationally. The ISO certification indicates an intensive process of
documenting and commitment to keeping procedural process documented. With the new
requirements to meet SOX government regulation, the ISO opportunities increase. While SOX is
about compliance, ISO is about international opportunities not otherwise available to corporations
when incorporating and DOCUMENTING Patch Management best practices you support the firm
in meeting long term ISO 900x compliance goals.
The world is becoming more and more global. Reviewing and understanding ISO can give the
corporation a HUGE competitive advantage and can open the doors to new opportunity otherwise
not possible. Consider adding an added WIN to your patch management best practices project.
Look into getting ISO certified, it is not just for manufacturing plants anymore.
The specific security guidelines of ISO17799 also provide a base for justification of patch and
software management even though the process is not expressly stated in the documents from the
ISO organization. The ultimate protection for vulnerability is to remove the vulnerability either in
the form of removing the service or, if you are unable to remove the service, ensure that the
flawed code is replaced by the patched ISO code. However, one should remember that while we
tend to stress “security patches”, patches can also be needed for fixing software bugs. Section
6.3.3 of the ISO 17799 standards require that procedures for reporting software malfunctions be
put in place.
Compliance with industry association standards and regulations
How can you reduce risk and increase compliance? Standards, Standards, Standards! There is
some thought that “there has been progress in developing information-security risk metrics over
Page
81
the past two decades, but there is still a way to go before standard metrics are established,
114
adopted, and practiced.”
The following is a list of a number of institutes who offer Security
standards.
The International Information Security Foundation (IISF) Generally Accepted Systems Security
Principles (GASSP) located at http://web.mit.edu/security/www/gassp1.html.
The International Standards Organization (ISO) ISO 17799 located at
http://www.iso.org/iso/en/ISOOnline.openerpage.
The European Information Security Forum (ISF) Standard of Good Practice located at
http://www.isfsecuritystandard.com/index_flash.htm
The Institute of Internal Auditors (IIA) Systems Assurance and Control (SAC) located at
http://www.theiia.org/esac/index.cfm.
The Information Security Audit and Control Association (ISACA) Control Objectives for
Information and Related Technology (CobIT) located at http://www.isaca.org/
How can we help the CxO or IT Technical director to summarize these standards and simplify the
process? There are a few fairly simply rules of thumb: Document processes so that if the person
responsible for a process is no longer available the business will continue. Update these
documents for change happens fast in the transitional phase of the global economy and certainly
in the world of technology culture growth. Create your standard so that it is a living document
and grows with the business. This requires that more than one individual understands and can
easily update said documentation. Define efficient processes including the process of pushing
out updates (including testing, backing up, and adopting change).
As a small business consultant, I recommend reviewing the following sample documentation and
guidance:
HIPAA: http://cms.hhs.gov/hipaa/hipaa2/readinesschklst.pdf
Gramm Leach Bliley: “Security Check: Reducing Risks to your Computer Systems:
http://www.ftc.gov/bcp/conline/pubs/buspubs/security.htm
Financial Institutions and Customer Data: “Complying with the Safeguards Rule,”
http://www.ftc.gov/bcp/conline/pubs/buspubs/safeguards.htm
ISO 17799: http://csrc.nist.gov/publications/secpubs/otherpubs/reviso-faq.pdf
Governmental engagements: http://csrc.nist.gov/pcig/cig.html
SOX 404 (for small companies that are subsidiaries of publicly traded companies)
http://www.pcps.org/pdf/article_mike_ramos_01.pdf
http://www.isaca.org/Template.cfm?Section=home&Template=/ContentManagement/Con
tentDisplay.cfm&ContentID=12406
Cross Border Privacy Issues:
http://www.itgi.org/Template_ITGI.cfm?Section=Security,_Control_and_Assurance&CON
TENTID=5556&TEMPLATE=/ContentManagement/ContentDisplay.cfm
EUROPA Data Protection Guide: http://europa.eu.int/comm/internal_market/privacy/guide_en.htm
Recent legislation in California, in the form of a law in effect as of January 1, 2005 called AB1950,
requires that firms take “reasonable security measures” to protect the confidential data of
California residents that could be labeled as “identity data.” I urge you to work closely with your
clients if they are in one of these industries or are having these issues. In addition, obtain the
resources you need, or find appropriate security consulting firms to help you stay informed.
Return on Investment (ROI)
What is the real return on investment to putting procedures in place? Why bother to document
“another” procedure and carefully outline processes? The question is easier answered than not,
but many CxO’s struggle with communicating the real understanding around the ROI of security.
“Unfortunately, relatively little attention has been paid to economics, and to the applied financial
practices that grow out of economics, when it comes to information security. In 2003, the annual
CSI/FBI Computer Crime and Security Survey reported average losses per respondent of about
Page
82
$800,000. In fact, the real headline should be that even those loss totals don't do justice to the
115
magnitude of information security crime and related costs.”
"Economics--not technology-determines what security technologies get used," says Bruce Schneier, security expert and
author of Beyond Fear: Thinking Sensibly about Security in an Uncertain World. "These days, I
116
feel like I do more economics than computer security."
“In the commercial world, the cost of a
117
cyber security breach is measured by both ’tangibles’ and ’intangibles’.”
Calculate the
tangibles based on estimates of:
• Lost business, due to unavailability of the breached information resources
• Lost business, that can be traced directly to accounts fleeing to a “safer” environment
• Lost productivity of the non-IT staff, who have to work in a degraded mode, or not work at
all, while the IT staff tries to contain and repair the breach
• Labor and material costs associated with the IT staff’s detection, containment, repair and
reconstitution of the breached resources
• Labor costs of the IT staff and legal costs associated with the collection of forensic
evidence and the prosecution of an attacker
• Public relations consulting costs, to prepare statements for the press, and answer
customer questions
• Increases in insurance premiums
• Costs of defending the company in any liability suits resulting from the breached
company’s failure to deliver assured information and services
Not all of these tangible costs will occur with each breach; some will only occur with major, wellpublicized breaches.
The intangibles refer to costs that are difficult to calculate because they are not directly
measurable, but are nevertheless very important for business. Many of these intangibles are
related to a “loss of competitive advantage” that results from the breach. For example, a breach
can affect an organization’s competitive edge through:
• Customers’ loss of trust in the organization
• Failure to win new accounts due to bad press associated with the breach
• Competitor’s access to confidential or proprietary information
“In the military, [we must also take into consideration] the tangible costs measured in human
lives, replacement costs of equipment, and prolonged military operations. The intangibles would
include loss of tactical advantage, loss of international prestige, and impaired negotiating
positions.” As Anita D’Amico points out, taking a close look at the break down in costs measured
by tangibles and intangibles helps to understand the cost of a missed patch that causes computer
down time. Certainly, security is not only about patches, but also by ignoring patch management
an avoidable security breach could occur. A survey of 882 respondents determined that the MS
Blaster worm cost $475,000 per company (median average - including hard, soft and productivity
costs) with larger node-count companies reporting losses up to $4,228,000.
MS Blaster is simply one example among many! We must then look at the constraints faced by
management in budgeting for security.
Page
83
From Information Security Magazine July 1999, "Top Obstacle is Budget: What is the SINGLE
greatest obstacle to achieving adequate info security at your organization?"
Consider these top obstacles to achieving adequate security and notice that right behind budget
constraints is “Lack of Senior Management Support.” "People in information security are often
technicians--gear heads," the report notes. Few IS managers have come up through the ranks of
118
accounting or financial management, so they don't think in [economic] terms,” the report adds.
It is critical that management focus on the economics of security and dare to go one step further
and recommend a CFO who is also well versed and experienced in technology and the security
niche. It is unfortunate, that more often than not it takes an event to occur before management
realizes that installing a patch management tool is of value.
What do we really need?
According to Kathryn Korostoff of Network World, “Network security is one of the hardest
technology categories for which to create an ROI analysis”119 but I doubt that it would be very
hard to find a number of other situations where the ROI relies on softer variables and, as such, is
as difficult. The goal is to break down the variables into manageable parts and then use the
resources of the web, human experience, and knowledge to put parameters around those
variables. Remember that sometimes these variables are very specific to the organization. When
we understand the business processes down to the tasks preformed by specific departments and
people, we can train staff to have an alternative path by default for completing the necessary
jobs.
If we look at a specific staff position and the task that position must accomplish we can determine
the impact of having limited or no access to the internet on the completion of these tasks. A sales
person making calls to prospects certainly needs access to customer relationship software, but
depending on their style and the software they depend on they might not need access to the
Page
84
internet. If we break down a number of sample components, we can make informed opinions and
plan responses to security breaches. We might even consider building these job descriptions into
the position statement stored with the HR department.
When considering the ROI around patch management best practices we really have three
projects. The first is that of creating and implementing a patch management best practice policy.
This roll out takes time and resources and it is important to include this initial investment in our
analysis. The second project is the ongoing maintenance and modification of the patch
management best practices procedures, including training new staff, maintaining vendor
relationships, and modifying documentation. Lastly, there is the project that defines and weighs
options when a breach occurs. This last project is often bundled with the corporation’s disaster
recovery planning or business continuity project. Here is an overview of some of the variables in
each project.
Defining the Patch Management Best Practices—First ROI Variables Project
In defining patch management best practices, we need to consider that the end goal is to have a
completely documented process that allows for continuous updates. This document should be a
living document that, once created, evolves into a Maintenance Project.
Intellectual teamwork—a team assembles the best practices. This team meets regularly to
discuss the project and to make plans for completion. Who is on the team? What credentials are
appropriate?
Research by individual team members—having complete data on the business environment,
the industry niche, and the tasks completed within the organization assures coverage of all
variables.
Decision-making takes time and input from the entire team. The decision-making component
might also need input from higher levels of management.
Documentation—writing, editing, typing, and binding the documentation require resources.
Review and acceptance—once the document is completed each department, the management
team, and perhaps even the board of directors needs to review and approve the procedures.
Software and hardware deployment—the actual analysis of deploying the above documented
best practices requires energy. A reality check, is it realistic? Who will analyze the costs of
needed components?
Industry specific variables can include other vendors who must approve the process such as
the corporate lawyer (for industry and contract compliance), or the corporate accountant (for SEC
compliance)
Actual ROI—the cost and return on investment analysis should be written and then kept with the
best practices patch management policy for any new manager or staff member exposed to the
original documentation. These variables change with time and part of project two will be looking
at the variables used at the outset. Some forward thinking firms have justified patch management
practices by identifying the costs saved by not hiring help desk staff, or used the process of asset
inventory to point out the unidentified risks and compliances issues.
Ongoing maintenance and modification—Second ROI Variables Project
Update schedule—when will the PMPB be updated and reviewed—quarterly, monthly, or
yearly?
Review team—who will do the review? Will there be a team, who will be on the team?
Research (continued)—who will do the follow-up research? Where will they start? What new
information will they look for? How do they know about this new information?
Documentation (updated)—who will do the actual writing and updating of the documentation?
How will he or she coordinate with the researcher? Who will manage the process?
On going relationship building with key vendor contacts—this takes time and is critical to a
best practice in this world of forever changing software and variables. Which people are involved?
How do they meet? When do they meet? Which process assures compliance with vendor
relationship policies?
Page
85
Hardware and software upgrade costs—change daily. What is the policy for upgrading
software and hardware? It may be necessary to implement more technology in order to provide
more tools for asset inventory and control.
Management—needs to set aside time to review the established procedures and policies.
Testing—lab equipment, space, licensing
Fire drill—does the firm support testing in a simulated situation a potential emergency.
When a Breach Occurs – Business Continuity—Third ROI Variables Project
Down time--what is the cost for your business to be offline? How long can you be offline?
Equipment replacement—if X percentage of equipment was damaged due to a disaster, what
would be the cost to replace it? How long would it take to replace? What would be the cost of that
time?
Backup location—if your location is destroyed, what is the cost, in time and dollars, to setup the
business in a temporary location?
Alternative power generator—if power is lost and all you need is power does it makes sense to
borrow, rent or purchase a generator?
Human impact (personnel loss that affects productivity)—even the best of us is not at our
best when someone close to us has been hurt. The impact on our best asset, the people within
the firm, plays a role in the ROI process.
This exercise of creating a list of variables and breaking down the tasks shows that doing an ROI
can be both overwhelming and very manageable. Certainly, the lists above do not include all
variables you must measure for but the goal was to get you thinking about others. Make it fun,
have a staff meeting where your only task is to brainstorm every possible variable appropriate to
each of the above three projects. Make a list and expand the above on an ongoing basis. Build
the game into the culture of the firm. Use magnetic words on the corporate refrigerator; give
monthly prizes to staff who think of something new. Prizes can be something such as a two-hour
lunch break, a gift certificate, or the first option on purchasing used equipment.
Once you have your list of variables, break each into as many parts as possible, within reason.
When you break down the ROI process into bite-size pieces, it is easy to assign values to the
different components. Instead of assigning values based on an economic model, you might
consider assigning points of importance. You can then assign energy to the items that rate the
highest on the points scale. This look at ROI takes you out of the limiting mindset of economics
and tangible items.
We work in a knowledge worker’s world. Valuing knowledge workers is an art most companies
have yet to master! How do I know this? Just look around at how many service companies bill by
the hour. Is measuring time truly the right variable for calculating costs? Many would argue a
resounding “No,” and the momentum is building. As such, how can we value the loss of time
strictly on that hourly model? If you have strong feelings about the billable hour and costing by
hour, check out Ron Baker’s books on the subject. He is a passionate speaker and writer and has
goals of eliminating the old industrial culture of the billable hour for new models based on the
knowledge!
The traditional risk analysis calculation also works as a basic budget analysis and ROI
calculation:
SLE x ARO = ALE where
SLE = Single Loss Expectancy
ARO = Annualized Rate of Occurrence
ALE = Annualized Loss Expectancy
The accounting staff, risk management staff, and information technology staff need to cooperate
on threat modeling your network. What are the biggest threats to your network? It is only in the
process of identifying those risk factors of your firm that you will be able to justify to management
that patch deployment is more than software deployment and asset management. As we go
Page
86
forward and patching is a key process of fixing and blocking security vulnerabilities, best practices
merely becomes part of compliance. Return on investment should reflect this “return” that
includes intangibles that cannot be easily quantified. History has shown us repeatedly that we do
not have control of our networks and that we cannot depend on our fragile “outside” perimeters
anymore. We must begin the process of adding layers and building up or hardening our interior
networks. Patching is one of the processes that assist in layering and hardening the defenses we
use to reduce risk.
Have we convinced you that patch management best practices are important? How can you as a
manager work with your IT team to roll out a patch management best practice? Or, if you are the
IT Manager how can you work with your management team to get the funding needed to make it
all happen? How can you get buy in? When technology runs and flows smoothly and systems
click along many benefits develop; however, what about the “human” factor associated with
effecting a change of even the smallest magnitude? Nothing motivates a manager to buy into a
concept quite like the thought of making more money and reducing personal inefficiencies that
compromise available time. The number one consideration is to know your audience! When you
work toward getting buy-in, you need to know what motivates your audience.
Sometimes this is money considerations, but sometimes it is one of the many other variables.
Knowing which variable to emphasize is the trick. For instance, for firms that make money by
selling time, saving time means making money and increasing the bottom line. You can also
consider the issue of compliance and risk. For firms where management is particularly exposed,
such as in publicly traded companies, reporting and the potential for analysis and proof that the
corporation is compliant can be the biggest motivating factor. Analysis can also be a factor for
helping to direct and focus a firm on its most profitable path to best practices. This reporting
component can reassure a CEO that his board of directors is not going to have problems with the
projects you are proposing. Direct reports also motivate your audience. Never forget that the
decisions in medium to large size firms rarely stop at one person. Speaking of one person, an
ROI must reflect individual style with appropriate personal and firm-wide emphasis. ROI
presentations vary widely depending on the audience. When you align your ROI with their goals,
you find solutions that meet both your needs.
Finding time in a crisis management environment
Who has time to spend hours and hours preparing ROI presentations for top managers? Doing so
takes time, but they can evolve by using bite-size pieces over time. Take the need to list all the
different variables affecting an ROI analysis. Split this project into 25 different subsets and then
assign each subset to a different person. You might ask the director of HR to make a list of all the
things affected by computer problems. You might ask the office administrator how a phone
system failure would affect clients. The marketing department is also a huge resource. They
understand how crisis and unavailability can affect customer relations. Do not forget about the
accounting department either. Many are numerically sensitive and can run some numbers for
you. Get people involved. Make your quick list and then delegate. The beauty of the computer
system is that it affects all departments and when considering variables for ROI analysis multiple
perspectives and diversification can be a huge asset.
Purchasing patch management tools
Patch management tools play a significant part in a patch management best practices policy and
in the efficiency of an IT department. These software products increase consistency and
compliance in the management of corporate software and hardware updates. People who work
with patch management software classify the patch management tools into two categories; the
tools are either “agent” or “agentless” applications. An agentless solution does not install any
software on the individual client computers. The agentless software can distribute or push
updates to multiple machines without needing to first install or find an “agent” on the receiving
machines. Agentless software rolls out in minutes as opposed to days. The benefits of an
Page
87
agentless solution are numerous, but there are also disadvantages. An agentless system is more
vulnerable. Many times agentless software will not comply with security policies that deal with
distribution of encrypted data. The benefit of an agent-based system is that software updates and
patches can be uniquely encrypted, delivered, and then unencrypted using the features of the
agent software. Unfortunately, agent based solutions also contain the agents, which are software
components that may also require updates, patches, and changes. Using an agent-based
solution increases the complexity and requirements on a machine-by-machine basis.
An extremely important component of patch management solutions is the reporting module.
When reviewing a tool, look carefully at the built-in, pre-configured reports. Patch management
software reporting can significantly decrease the amount of due diligence that must be done on
an ongoing basis. The reporting component of patch management software can also indicate
which patches need immediate deployment, which can wait, and which are unnecessary.
Sophisticated patch management software packages offer robust reporting modules.
Sophistication in patch management software also shows itself in support for multiple platforms
because a patch management tool in an enterprise environment often needs to recognize and
resolve patching on multiple operating systems and platforms. Even in small environments, there
are more and more alternative operating systems and platforms widely available. Consider the
mix of Open Source solutions and Microsoft Solutions. Many smaller companies find that having
a mixed environment offers more options for less money.
Patching the operating system is one consideration, but we also run numerous third party
programs and each of these third party programs also has patches. The ability to recognize third
party programs and pull patches from a wide variety of web resources is also a sign of a robust
patch management solution. Here is a checklist of things to ask questions about when
researching patch management software.
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
Agent vs. agentless
Reporting: pre-defined and custom options
Rights Management: Can access to the toolset be restricted and to what level?
How many clients can the package support?
Are there customers using it with this many clients?
What are the biggest and smallest installations?
Does it depend on a database back-end engine? Which one?
What operating systems are supported?
What third party applications are supported?
Is the list of third party applications updated regularly? When?
Does it cost to get the new libraries of supported third party applications?
Is the User interface easy to use?
Does the user interface require technical training?
Does the application require training? If yes, how much?
Are there other features not related to patch management built into the core package?
In general, just as with any other due diligence review of a new software package, you want to
ask as many questions as possible. A great resource for posting questions is the
patchmanagement.org virtual community, but many of the security “centric” virtual communities
also offer independent perspectives from people who have experience with many of the patch
management tools available today.
Vendor Issues
Controlling the patch management process requires significant relationships with companies
involved in the release of equipment and software. We stressed this earlier. In the current
technology environment, a software company generally takes responsibility for their software and
releases regular updates based on their methods of operation. For your company to reduce risk
Page
88
and exposure it is important to understand the models in which your vendors operate. These
models should also be part of the consideration process when choosing to work with a particular
software or hardware vendor. Allowing vendors to manage patch management processes may
compromise your work to maintain and stay within a high quality patch management best
practices system. The key is to understand the vendor’s business model and then design your
system around their offerings. If their offerings do not fit a good patch management best practice
then consider working with a vendor who is more agile and responsive.
In addition to picking vendors based on agility, you also want to create strong bridges of
communication between multiple people within the vendor firm and with multiple people within
your firm. Vendors can and have released incompatible patches, they can and have struggled
with timing and delay issues based on their own testing schedules and their vendor relations and
they can also have policies in place that could bind your hand. The software and hardware
vendor have the power to threaten or even void service contracts if you touch or update their
software or hardware.
Conclusion
As much as the world of security is becoming more important for everyone to understand, it is still
a technical niche with its own language and unique set of acronyms. Extensive resources are
available. This text helps focus management and staff on the three critical components of
implementing and rolling out good patch management best practices. Each of these three
components functions as a unique project. The first is accepting, deciding and actually rolling out
a patch management best practice policy; the second is the project of maintaining and updating
the patch management best practices policies so that they are viable on a continued basis; and
the third and perhaps the most well known is building disaster recovery planning into the
corporate culture. Patch Management is about reducing risk, increasing continuity, expanding
options and relationships so that when a crisis occurs, everyone not only knows where to turn,
but who to turn to for help and resources.
Page
89
Appendix A– Patch Testing
Testing plans
9 Identify the operating system being tested
9 Identify the Service pack level
9 Identify the hotfixes installed on your systems [if in addition to security fixes]
9 Identify critical third party applications
9 Identify third party applications that have had patching historically
9 Identify those files used in patches that may have causes issues in the past. Are the
included in this current patch? Assign testing resources appropriately.
9 Determine if the bulletin accurately describes patch deployment. If not, determine if
additional resources for testing or imaging need to be in place before approving the
patch.
9 Test the installation of the patch both manually and via your automated patch technology.
9 Can you uninstall the patch using add/remove program or your patch tool.
9 Review the processes of your line of business applications. Are they performing as
expected?
9 Attempt to replicate a production environment by using imaged data. Testing tools may
assist in attempting to replicate your production system but are not a substitute for an
image of your system.
9 Stress tools to test a Web server:
o http://support.microsoft.com/default.aspx?scid=kb;EN-US;q231282
9 Download details: Windows Application Compatibility Toolkit 3.0:
http://www.microsoft.com/downloads/details.aspx?FamilyID=7fc46855-b8a4-46cd-a2363159970fde94&DisplayLang=en
120
9 Set up performance and monitoring tools to review your testing machines
such as
121
PerfMon, tools from Sysinternals
and review all log files.
9 Confirm the installation of the patches via registry review or other means
Page
90
Appendix B—Sample budget request
CONCLUSION AND RECOMMENDATION: Recommendation to the board to approve the
purchase and installation of software patch management software to remotely manage all of the
computing devices in the network as deemed by the IT staff. The estimated cost of this project
will be in the range of ___________ to ____________.
EXECUTIVE SUMMARY: The computer assets of the firm are an integral part of the revenue
generation of the company. At the present time, unnecessary IT staff time is spent in attempting
to control, administer and apply software patches to our current network system. We are
requesting that the firm implement a more structured approach to software patching and
management of the computer systems in the firm.
BACKGROUND: Over the years through normal expansion and firm mergers and acquisitions,
the network of the company has grown organically thus not providing the proper infrastructure to
allow for the IT staff to properly control and manage the computer systems of the company,
Currently the lack of software patching tools and lack of remote connectivity and control is
contributing to the costs of the helpdesk in the firm. The following techniques will help reduce
these costs:
Standardization of desktop and server hardware and software--aids in the process of more
control of the change management process we recommend that an inventory of all assets be
undertaken and documentation of approved applications be prepared and only these allowed
applications be installed in the firm.
A virtual lab replicating these standardized desktop and server installations—aids in the quicker
and faster testing of patches to deploy them quickly to the machines identified as needing these
patches.
Purchase of patch management software and/or tools to better aid in software patch application—
currently the IT staff does not have the necessary tools to quickly and easily provide reports to
management to ensure compliance with the firm’s minimum security policies.
FISCAL IMPACT: Initial costs of _________ and annual maintenance fees of __________ are
expected to be incurred in hard costs and staff costs are expected to range from _________ to
___________ in the testing and implementation of this solution.
Page
91
Glossary
Application - A program or group of programs designed for end users. Software can be divided
into two general classes: systems software and applications software. Systems software consists
of low-level programs that interact with the computer at a very basic level. This includes operating
systems, compilers, and utilities for managing computer resources. In contrast, applications
software (also called end-user programs) includes database programs, word processors, and
spreadsheets. Figuratively speaking, applications software sits on top of systems software
because it is unable to run without the operating system and system utilities.
Critical Update – this is not a security update, but a fix for an issue in broadly applied software.
They are publicly available and have an accompanying knowledge base article.
Driver Update – The update in a software component that supports and controls hardware Driver updates may come from the software vendor or they may be from the hardware vendor.
Feature pack – a software package that is released that includes non-critical additions to the
base software program. It is typically distributed between major releases.
Firewall - A system designed to prevent unauthorized access to or from a private network.
Firewalls can be implemented in both hardware and software, or a combination of both. Firewalls
are frequently used to prevent unauthorized Internet users from accessing private networks
connected to the Internet, especially intranets. All messages entering or leaving the intranet pass
through the firewall, which examines each message and blocks those that do not meet the
specified security criteria
Hotfixes – hotfixes are patches built to address specific issues. The Microsoft Hotfixes cannot be
distributed outside of your organization without written authorization from Microsoft. They are
obtained for FREE by calling Microsoft Product Support Services. They are not subjected to the
regular testing process.
Patch – “A piece of code added to software in order to fix a bug, especially as a temporary
correction between two releases.”
Patch Management - the act, manner or practice of managing, handling, supervising and
controlling the application of code to software in order to fix a bug, especially as a temporary
correction between two releases.
Security (data) – A way of insuring data on a network is protected from unauthorized use.
Service Packs (SP) – are cumulative packages of hotfixes, security updates, critical updates and
updates. The SP is tested both internally and externally.
Security Update – this is what we traditionally refer to in the existing patch management circles.
There is a severity rating included in the updates and they are always accompanied by a Security
bulletin discussing the issue and a Knowledge base article that describes the patch in more
detail.
Update – fixes a non-critical, non-security related issue.
Update rollup – cumulative package of hotfixes, security updates, critical updates, and updates.
This is tested for easy deployment.
Upgrade – software that updates and upgrades a piece of software to a newer version while
keeping the settings and data from the prior program
Page
92
Notes
Chapter 1—What Is Patch Management
1
“Analysis of the Sapphire Worm - A joint effort of CAIDA, ICSI, Silicon Defense, UC Berkeley EECS and
UC San Diego CSE,” retrieved August 29, 2004 from http://www.caida.org/analysis/security/sapphire/
2
“Threats and Countermeasures: Security Settings in Windows Server 2003 and Windows XP,” (Redmond,
WA: Microsoft, Inc., 2004) from http://www.microsoft.com/downloads/details.aspx?FamilyId=1B6ACF93147A-4481-9346-F93A4081EEA8&displaylang=en
3
Allen, Julia H. The CERT Guide to System and Network Security Practices, New York: Pearson Education,
2001.
4
“Description of the standard terminology that is used to describe Microsoft software updates,” (Redmond,
WA: Microsoft, Inc., 2004) from http://support.microsoft.com/default.aspx?kbid=824684
5
Ferguson, Niels and Bruce Schneier Practical Cryptography New York: John Wiley & Sons, Inc., 2003.
6
“10-Step Emergency Response Plan for Security Attacks” Redmond, WA: Microsoft Corporation, 2004 from
http://members.microsoft.com/Partner/Campaign/SecurityOutreach/sell/10StepEmergencyResponsePlanforSecurityAttacks.pdf
7
“Upcoming Advisories” Aliso Viejo, CA: eEye Digital Security, 2004 from
http://www.eeye.com/html/research/upcoming/index.html
8
http://lists.netsys.com/mailman/listinfo/full-disclosure
9
Swiderski, Frank and Window Snyder Threat Modeling Redmond, WA: Microsoft Press 2004.
10
While we will discuss alternative OS, due to market share and desktop impact, the bulk of the discussion
will surround the Microsoft operating systems.
11
“Description of the Contents of a Windows Server 2003 Product Update Package” Redmond, WA:
Microsoft Corporation, 2004 http://support.microsoft.com/default.aspx?scid=kb;en-us;824994
12
“Microsoft Security Bulletin MS04-022” Redmond, WA: Microsoft Corporation, 2004
http://www.microsoft.com/technet/security/bulletin/MS04-022.mspx
13
Koetzle, Laura, Ted Schadler, Charles Rutstein, and Robert Whiteley Can Microsoft Be Secure?
Cambridge, MA: Forrester Research, March 2003
http://www.forrester.com/ER/Research/Report/0,1338,16275,FF.html
14
For purposes of this document we use only RedHat’s Enterprise Linux distribution in the examples;
however all of the Linux distributions provide similar services.
Chapter 2-- Change Management and Risk Analysis
15
Farrell, Nick. “SQL Slammer hits Microsoft,” vnunet.com, January 28, 2003; found at
www.networkitweek.co.uk/news/1138312.
16
“Security Configuration Guides,” found at www.nsa.gov/snac/index.cfm?MenuID=scg10.3.1
17
“CIS Benchmarks/Security Tools,” found at www.cisecurity.org/
18
“Reducing Over 80% of Windows 2000 Professional Vulnerabilities with the Consensus Security
Benchmark Settings,” Center for Internet Security, found at
www.cisecurity.org/Documents/Reducing_Over_80.htm
19
Shimonsky, Robert J., “Risk Assessment and Threat Identification,” Windows Security.com, August 29,
2004, found at www.windowsecurity.com/articles/Risk_Assessment_and_Threat_Identification.html
20
“RFC 1157 - Simple Network Management Protocol (SNMP),” found at
http://www.faqs.org/rfcs/rfc1157.html; “Using SNMP for Network Management,” Windows NT Resource Kit,
found at http://www.microsoft.com/resources/documentation/windowsnt/4/server/reskit/enus/net/sur_snmp.mspx
21
VMware, Virtual infrastructure, http://www.vmware.com/ , Microsoft Virtual PC
http://www.microsoft.com/windows/virtualpc/default.mspx, and Microsoft Virtual Server
http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx
22
Johansson, Jesper, “Oh Patch, How I hate thee, let me count the ways,” Microsoft Corporation, found at
http://www.microsoft.com/technet/community/columns/secmgmt/sm0404.mspx
23
Boyle, Neil, “SQL Server Security Checklist,” Database Journal, June 13, 2003, found at
http://www.databasejournal.com/features/mssql/article.php/2221471
Chapter 3—It’s all about risk
24
Security bulletin 03-029 retrieved from http://www.microsoft.com/technet/security/bulletin/MS03-029.mspx
Page
93
25
Schultze, Eric, Microsoft’s 2004 Security report card, found at
http://www.ftponline.com/wss/2004_08/magazine/departments/guestop/
26
Demanding more from vendors, Rain Forest Puppy as extracted and placed on the following web site:
http://www.sbslinks.com/really.htm
27
http://blogs.msdn.com/aaron_margosis/archive/2004/06/17/157962.aspx
28
Microsoft rating of Security Bulletins, http://www.microsoft.com/technet/security/bulletin/rating.mspx
29
File corruptions in AutoCAD files after the installation of Security bulletin 03-026,
http://usa.autodesk.com/adsk/servlet/ps/item?siteID=123112&id=3431487&linkID=2476435
30
Technical information for Security bulletin 03-026 - http://support.microsoft.com/?kbid=824136
31
Keep in mind that at that time of these historical bulletins, you had to review the underlying knowledge
base article; currently bulletins now include the file versions and detailed information on the security bulletin
itself.
32
Patch Management listserve located at http://www.patchmanagement.org/
33
Patchtalk listserve located at http://www.patchtalk.com/
34
Microsoft online communities resource page - http://www.microsoft.com/communities/default.mspx
35
Workgroup for Electronic Data Interchange listserves http://www.wedi.org/listserve/
36
HD Moore’s Metaspoint project to allow for more controlled exploit testing - http://www.metasploit.com/
37
My analysis of the timeline between the patch release of 03-026 to the web and the resulting release of
exploit code to the web - http://www.sbslinks.com/timeline.htm
38
http://www.microsoft.com/technet/security/topics/patch/stdpatex.mspx
39
http://www.sysinternals.com/ntw2k/freeware/listdlls.shtml
40
http://www.microsoft.com/technet/desktopdeployment/depprocess/depprocessris.mspx
41
http://singe.rucus.net/blog/archives/243-Sun-Recommended-Patch-Management-Policy.html
42
http://docs-pdf.sun.com/817-0574-12/817-0574-12.pdf?biga=15
43
http://www.microsoft.com/resources/documentation/WindowsServ/2003/standard/proddocs/enus/Default.asp?url=/resources/documentation/WindowsServ/2003/standard/proddocs/enus/SAG_MPmonperf_06.asp
44
http://www.sysinternals.com/
45
Infraguard (2004) Technology Risk Checklist retrieved October 10, 2004 from
http://www.infragard.net/library/pdfs/technologyrisklist.pdf
Chapter 4—Are we ready to patch yet?
46
48
http://www.helpdesk.com/software-helpdesk.htm
49
http://support.microsoft.com/default.aspx?pr=securityitpro
50
http://bugzilla.redhat.com/bugzilla/query.cgi
51
http://www.ftponline.com/wss/2004_08/magazine/departments/guestop/
52
http://www.sans.org/resources/policies/
53
http://www.sans.org/resources/policies/Server_Security_Policy.doc
54
http://www.sans.org/resources/policies/Aquisition_Assessment_Policy.doc
55
http://www.sans.org/resources/policies/Audit_Policy.doc
56
http://www.sans.org/resources/policies/Virtual_Private_Network.doc
57
http://www.sans.org/resources/policies/
58
http://www.cisco.com/application/pdf/en/us/guest/netsol/ns466/c664/cdccont_0900aecd800ce939.pdf
59
http://www.microsoft.com/downloads/details.aspx?FamilyID=fe902704-52dd-4bbe-8a75f8fbb76cd28a&DisplayLang=en
60
http://searchnetworking.techtarget.com/originalContent/0,289142,sid7_gci992761,00.html
61
http://download.microsoft.com/download/0/7/e/07ed1953-0ab5-41ea-b5da-41cf8bb9cdae/Quarantine.doc
62
http://www.microsoft.com/technet/community/tnradio/archive/tntrans_01.mspx
63
http://www.sysinternals.com/ntw2k/freeware/listdlls.shtml
64
http://www.sysinternals.com/ntw2k/freeware/procexp.shtml
65
http://dotnet.org.za/kevint/archive/2004/04/01/931.aspx
66
http://www.google.com/
67
http://www.google.com/grphp?hl=en&tab=wg&q=
68
http://www.microsoft.com/downloads/details.aspx?FamilyID=cebf3c7c-7ca5-408f-88b7f9c79b7306c0&DisplayLang=en
69
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
47
Page
94
70
http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96eeb18c4790cffd&displaylang=en
71
http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/xpsp2man.mspx
72
http://download.microsoft.com/download/5/d/b/5db098e2-43d3-48bc-9e5be72ab9c65c61/IPSec_Server2003.doc
73
74
75
76
http://www.microsoft.com/technet/security/topics/patch/stdpatex.mspx
77
http://www.sbslinks.com/really.htm
78
http://info.sen.ca.gov/pub/01-02/bill/sen/sb_1351-1400/sb_1386_bill_20020926_chaptered.html
79
http://www.cio.com/archive/021502/security.html
80
http://www.sims.berkeley.edu/resources/affiliates/workshops/econsecurity/econws/06.doc
81
http://www.cert.org/nav/index_purple.html
82
http://isc.sans.org//survivalhistory.php
Chapter 5—Life beyond patching Microsoft
83
http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/gpdepsp2.mspx
Find more information on the Group Policy Management Console at
http://download.microsoft.com/download/a/9/c/a9c0f2b8-4803-4d63-8c323040d76aa98d/GPMC_Administering.doc and
http://www.microsoft.com/windowsserver2003/gpmc/default.mspx
85
http://rhn.redhat.com/errata/RHSA-2003-280.html
86
http://www.unixwiz.net/techtips/building-source.html
87
http://docs-pdf.sun.com/817-0574-12/817-0574-12.pdf?biga=15
84
Chapter 6—Justifying Patch Management—A Business Tutorial
88
Gartner (2003, 5) 2003 Press Release retrieved October 10, 2004 from
http://www3.gartner.com/5_about/press_releases/pr29may2003a.jsp
89
Smith, R (2003) "The Chief Technology Officer: Strategic Responsibilities and Relationships", Research
Technology Management, July-August, 2003. and Smith, R (2003) "Maximizing the CTO's Contribution to
Innovation and Growth", retrieved October 5, 2004 from www.CTOnet.org
90
American Institute of Certified Public Accountants, AICPA (2004) CPA Financial Manager's Role in a
Company retrieved May 23, 2004 from http://www.aicpa.org/cefm/CPA_fincl_mgr_role.asp
91
Gartner
92
Gartner
93
Messner, E (2004, 7) “Rx for Patching Mired in Red Tape” Copyright, 1994-2004 Network World, Inc.
retrieved September 18, 2004 from http://www.nwfusion.com/news/2004/070504hospitalpatch.html
94
Stoneburner, G, Goguen, A and Feringa, A (2002, July)The National Institute of Standards and
Technology (NIST) Special Publication 800-30 Risk Management Guide for Information Technology
Systems retrieved October 17, 2004 from http://csrc.nist.gov/publications/nistpubs/800-30/sp800-30.pdf
95
Stoneburner
96
Stoneburner
97
Stoneburner
98
Stoneburner
99
Stoneburner
100
Stoneburner
101
Cavitt, L (2004) “Questions on HIPAA and FDA approval of Biomedical devices” Online posting
September 3, 2004 WEDI SNIP Security Workgroup List http://subscribe.wedi.org
102
Messner, E (2004, 7) “Rx for Patching Mired in Red Tape” Copyright, 1994-2004 Network World, Inc.
retrieved September 18, 2004 from http://www.nwfusion.com/news/2004/070504hospitalpatch.html
103
Secretariat: NEMA (National Electrical Manufacturers Association) www.nema.org/medical retrieved
October 10, 2004 from http://www.nema.org/DocUploads/50496076-20B1-4DADA81B8B82F808B83E/medical-defending.pdf
104
E-Health Insider is managed and maintained by E-Health-Media.com http://www.e-health-media.com/
105
Messner, E
Page
95
106
AT&T News Release (1998, 11) AT&T Announces Cause of Frame-relay Network Outage retrieved
December 26, 2004 from http://www.att.com/news/0498/980422.bsb.html
107
Agilent Technologies (2002, 3) Preventing IP Network Service Outages retrieved December 26, 2004
from http://advanced.comms.agilent.com/n2x/docs/whitepapers/pdfs/sp-insight.pdf
108
Schmerken, I (2004, September 21) “Innovation In Motion” Publisher: Wall Street Technology retrieved
October 17, 2004 from
http://www.wallstreetandtech.com/story/riskManagement/showArticle.jhtml?articleID=47900742
109
Schmerken
110
Schmerken
111
Hicks, M (2004, October) PayPal returning to Normal eWeek retrieved October 17, 2004 from
http://www.eweek.com/article2/0,1759,1676610,00.asp
112
Kellermann, Sr, T (2003, June). Data Risk Management Specialist, Integrator Unit, The World Bank
retrieved October 17, 2004 from http://www.cyberpartnership.org/World%20Bank%20Erisk%20Heat%20Map.pdf
113
AICPA Business & Industry Internal Control Task Force (2003) Key Issues for Management - AICPA
Implementation Guidance and Tools for Sarbanes Oxley Key Internal Control Issues for Management
retrieved October 3, 2004 from http://www.aicpa.org/download/sarbanes/key%20issues%20document%20%20FINAL.pdf
114
Ozier, W (2003, May) Risk Metrics for IT Security retrieved October 3, 2004 from
http://www.securitypronews.com/securitypronews2420030805RiskMetricsNeededforITSecurity.html
115
Gordon, L and Richardson, R (2004, April) Economics of Cybercrime Computer Crime Research Center
retrieved October 22, 2004 from http://www.crime-research.org/articles/Economics_Cybercrime/
116
Gordon
117
D’Amico, Anita D. Ph.D. (2000, 9) What Does a Computer Security Breach Really Cost? Retrieved
October 10, 2004 from http://www.securedecisions.com/documents/CostsOfBreaches-SANSInstitute.doc
118
Gordon
119
Korostoff, K (2003, 8) The ROI of network security Network World, 08/25/03 retrieved November 29,
2004 from http://www.nwfusion.com/techinsider/2003/0825techinsiderroi.html
120
Microsoft (2004) Windows Server retrieved October 10, 2004 from
http://www.microsoft.com/resources/documentation/WindowsServ/2003/standard/proddocs/enus/Default.asp?url=/resources/documentation/WindowsServ/2003/standard/proddocs/enus/SAG_MPmonperf_06.asp
121
Sysinternals (2004) retrieved November 10, 2004 from http://www.sysinternals.com/
Page
96
In the Real World...
IT managers want easy to install and easy to use management software
that fits within their budget and delivers immediate value right out of the box.
…That’s the Ecora promise.
Ecora provides total configuration management solutions that automate multi-platform configuration
reporting, change monitoring, and patch management. Ecora's solutions enhance efficiency and reduce the
costs associated with IT compliance, business continuity, and vulnerability assessment while providing the
means to monitor change and plan for recovery. Cost effective, easy to implement, and easy to manage IT
solutions.
Ecora Patch Manager – Don’t let researching and manual patching take over your time – Know You’re
Patched, Today! Save time and reduce costs with a focused patch management solution featuring an
optional-agent technology for the automated deployment of critical patches and unique reporting capabilities
for both, IT administrators and managers.
Ecora Enterprise Auditor - An indispensable tool for documenting and managing your IT environment. If
you’re addressing Sarbanes-Oxley, HIPAA, CFR Part 11, or other regulatory compliance acts, Ecora
automates and delivers audit-ready reports that improve accuracy and, save time and money. Ecora
supports enterprise platforms: Cisco Systems, Lotus Domino, Microsoft, Novell, Oracle, HP-UX, IBM, Sun
Microsystems, Red Hat, and Citrix.
Ecora Dr. Wi-Fi - Continuously monitor the availability and performance of mission critical Wi-Fi networks.
Dr. Wi-Fi provides IT managers real-time data with easy-to-read “dashboard” reports and proactive alerts.
Ecora DeviceLock - Prevent users with USB drives from stealing your data. Halt unauthorized Wi-Fi
networks from gaining access to your valuable information and manage user access to devices.
Solutions for Managing IT in the Real World.
For more information about Ecora Software
www.ecora.com or 1.877.923.2672
Notice to Readers
The Patch Management Reference Guide does not represent an official position of Ecora
Software, The Norwich Group or Tamiyasu, Smith, Horn And Braun Accountancy Corporation
and it is distributed with the understanding that the authors and publisher are not rendering legal,
accounting, or other professional services in this publication. If legal advice or other expert
assistance is required, the services of a competent professional should be sought.
Patch Management Best Practices
Copyright © 2005 by Ecora Software Corporation
All rights reserved. For information about the procedure for requesting permission to make copies
of any part of this work, please call the Ecora Software Copyright Permissions Hotline at
603.436.1616. Otherwise, requests should be written and mailed to the Permissions Department,
Ecora Software
About the Authors
Susan E. Bradley, CPA, CITP, MCP, GSEC is a Microsoft Most Valuable Professional for Small Business Server and
Security. She is Chairman of the Technology committee of the California Society of Certified Public Accountants,
Chairman of the Top Technologies committee for the American Institute of Certified Public Accountants (AICPA), and
Committee Organizing member and speaker at the American Institute of Certified Public Accountants Tech 2003 and
Tech 2004 Conferences. She has written numerous articles on computers and security for the AICPA InfoTech newsletter,
the FCG Buzz newsletter, and the California CPA magazine, and is co-author of an upcoming book. She maintains a blog
on security topics and issues affecting the Microsoft Small Business Server 2003 platform at
http://www.msmvps.com/bradley. Reach her at sbradcpa@pacbell.net but if she doesn’t get back to you right away it’s
because she is sorting through all of the out-of-office messages she gets when she posts to the patch management
mailing list. Sign up at http://www.patchmanagement.org/. She is a principal with Tamiyasu, Smith, Horn and Braun in
Fresno, Calif. She writes an ongoing column, Ebitz, for AICPA’s InfoTech Update newsletter, and is chair of the Top
Technologies Task Force.
Anne A. Stanton, an enthusiastic proponent of networking, Anne A. Stanton is a consultant to consultants, specifically
in the Accounting and IT consulting niches. She creates bridges between business and technology. Anne is a member of
the AICPA's Top 10 Technologies Task Force, and an active participant in numerous forums including the Certified
Information Technology Professionals (CITP) discussion forum, the AICPA Technology Conference Chat user's group, the
Association for Accounting Marketers user's group, and the CPA Map group (CPAs discussing issues relating to the
management of their practices). She also moderates the NH/VT Upper Valley Microsoft Consultant’s group and is the comoderator of the InterNETnational Microsoft SBS Leaders Group. She is a contributing writer for Accounting Software
411.com, the CPA Technology Advisor, CPA Magazine and the AICPA’s IT Section Newsletter, InfoTech Update, as well
as many other industry specific journals. She is an active speaker and has presented at SMB Nation, the AICPA
Technology Conference, the OK State CPA State Society conference, and the New York Technology Assurance meeting.
She is the editor of the Information Technology Alliance newsletter and an active participant in numerous events
sponsored by the AICPA, state CPA societies and the Information Technology Alliance. Reach her at
astanton@thenorwichgroup.com.

patch management best practices

Transcription

Similar documents

bachmaster unit 2 patches

Chattanooga company builds brains behind tomorrow`s

Semco® Click-Patch Kit™ Instruction Sheet

Patching Instructions - Eye Care San Luis Obispo

march 2013 - Big Country Energy Services

Score (excerpt - pages 1/2)

Transdermal delivery of macromolecules y using Macroflux

Welcome to the Catkiller Store! C atkiller Sew on P atches C atkiller

Japanese topical patch formulations: What can we learn?

FlexTherm – Insulated Curtain Walls