“A security advisory was just published! Should I hurry and upgrade all my Cisco devices now?”
This is a question that I am being asked by customers on a regular basis. In fact, I am also asked why there are so many security vulnerability advisories. To start with the second question: Cisco is committed to protecting customers by sharing critical security-related information in a very transparent way. Even if security vulnerabilities are found internally, the Cisco Product Security Incident Response Team (PSIRT) – which is my team – investigates, drives to resolution, and discloses such vulnerabilities. To quickly answer the first question, don’t panic, as you may not have to immediately upgrade your device. However, in this article I will discuss some of the guidelines and best practices for responding to Cisco security vulnerability reports.
For the purpose of this article, I will explain how customers can respond to Cisco Security Advisories and Cisco Security Notices, as documented in our security vulnerability policy:
Cisco Security Advisories provide detailed information about significant security issues that directly involve Cisco products and require an upgrade, fix, or other customer action.
Cisco Security Notices document low- and medium-severity security vulnerabilities that directly involve Cisco products but do not warrant the visibility of a Cisco Security Advisory.
Cisco PSIRT scores and provides a risk analysis for each vulnerability or security issue that is investigated and disclosed using the Common Vulnerability Scoring System (CVSS) version 2.0. CVSS is a standards-based scoring method that conveys vulnerability severity and helps organizations determine the urgency and priority of a response.
Tip: Jean Reese, Sr. Manager of Cisco’s PSIRT and the Security Technology and Assessment Team (STAT), explained the differences between Cisco Security Advisories and Security Notices in a previousblog post.
Cisco’s Disclosure Cycle
Cisco generally discloses Cisco Security Advisories at on any given Wednesday. Cisco also releases two scheduled Cisco IOS Software bundles each year, as documented in our security vulnerability policy:
In direct response to customer feedback, Cisco releases bundles of Cisco IOS Software Security Advisories on the fourth Wednesday in March and September each year. This schedule applies to the disclosure of Cisco IOS Software vulnerabilities and does not apply to the disclosure of vulnerabilities in other Cisco products.
The following figure illustrates the Cisco IOS Software security advisory bundle timeline:
For all other products Cisco typically publishes vulnerabilities on any given Wednesday at 16:00 UTC.
Note: In some circumstances, Cisco PSIRT may publish vulnerabilities on other days of the week. For example, when coordinating the disclosure of industry-wide or third-party incidents and vulnerabilities.
Vulnerability Assessment and Patch Management
There is no “one-size-fits-all” patch management technique. Cisco has a very diverse set of products. Cisco’s product portfolio includes small business products to large service provider high-end routing products, from collaboration products to data center and video-related products, and much more. Security Advisories can include vulnerabilities in any Cisco product. The patch methodology can vary wildly as well.
Some patches may require a device reload (for example, in the case of Cisco IOS devices, Cisco ASA, etc.) and in others a reload may not be required (for example, in several cases with Cisco IOS XR Software, network management applications, and others). Everyone understands that patch management is a critical issue. We also understand that every organization must create a consistent environment that is patched or configured against known vulnerabilities. Unfortunately, good and practical solutions aren’t applied as often as many of us think (or hope).
Let’s review an example of an overall patch management process. By understanding the process you can then collect operational metrics that can help you measure success or identify gaps in such processes. The following figure summarizes a typical patch management process in a high-level perspective.
The following are the steps outlined in the previous figure.
Step 1: The vendor identifies and announces a vulnerability in a product. In the case of Cisco, we announce and publish all of our vulnerabilities on the Cisco Security Intelligence Operations portal.
Tip: You should always keep up with the vulnerability announcements from vendors (including Cisco). You can do this by subscribing to RSS, through CVE announcements, monitoring aliases such asBugtraq, or any other mechanism that vendors use to notify their customers that a vulnerability exists.
Step 2: You must have a good understanding of what devices are affected by such a vulnerability within your network. You can do this by manually looking at the affected platforms and software versions in each Security Advisory or by leveraging Cisco’s machine readable content to automate this process (this automation will be explained later in this article).
Step 3: It is possible that the vendor (in this case, Cisco) has identified workarounds that you can implement quickly. You need to identify those workarounds and understand how to apply them in your environment. Therefore, look for any workarounds by following recommendations within the “Workaround” section on each advisory or by consulting any Applied Mitigation Bulletins (AMBs) that may be published as a companion of the security advisory. AMBs include techniques that use Cisco networking devices to detect and mitigate known vulnerabilities.
Step 4: While a workaround is being placed, obtain the patch or fix for that vulnerability.
Step 5: In most cases you do not have a lot of time to test the new software before you deploy that patch or fix in your network. However, once the patch or image is certified, schedule a maintenance window to roll it out into the production environment.
Step 6: Apply the patch/software upgrade.
Identifying Affected Devices
Most network administrators and security professionals have many systems to patch and configure securely, with numerous versions of software and features enabled. Therefore, they are seeking ways to leverage standards and available tools to reduce the complexity and time necessary to respond to security advisories, assess their devices, and ensure compliance so they can allocate resources to focus on other areas of their network and security infrastructure.
Automating Vulnerability Assessment and Vulnerable Device Identification
In order to help customers respond to security advisories more effectively, Cisco PSIRT is now includingOpen Vulnerability and Assessment Language (OVAL) definitions in Cisco IOS Security Advisories, and also publishing Common Vulnerability Reporting Framework (CVRF) content for all Security Advisories.
OVAL and CVRF allow vendors to publish security advisories in an XML format intended for the sharing of security-related information in a machine-readable format.
OVAL is a standard designed to assist security administrators by accelerating the process of analyzing a system for the presence of a vulnerability or for configuration best practices. OVAL speeds up the assessment and processing of security-related information. Using OVAL, security administrators and other users can accelerate the process of detecting software vulnerabilities in Cisco IOS Software. OVAL content (often called “definitions”) can be downloaded directly from Cisco IOS Security Advisories. Each Cisco IOS Security Advisory includes a link to the corresponding OVAL definition(s).
Note: CVRF has been designed by the Industry Consortium for Advancement of Security on the Internet (ICASI), of which Cisco is a member and took a major role in its development. Please refer to the blog posts titled “The Missing Manual: CVRF 1.1 Part 1 and Part 2 for detailed information about CVRF. Additionally, please refer to the article I published a few months ago titled “Help! I Need to Respond to All These Cisco IOS Software Vulnerabilities and I Cannot Scale!!! to obtain more information about theSecurity Content Automation Protocol (SCAP) and OVAL.
Identifying Workarounds and Network-based Mitigations
In some instances, in-device workarounds are not available or can be very disruptive. However, there may be mitigation and identification techniques that can be deployed on infrastructure devices. AMBs, created by Cisco’s Applied Security Intelligence team, are companion documents to Security Advisories and other industry events that provide identification and mitigation techniques that administrators can deploy on Cisco network devices. For instance, Cisco network infrastructure devices can provide several countermeasures for exploit prevention, including features such as:
Each AMB includes detailed information about these techniques and many others. You may ask, “but where should these mitigation and visibility techniques be applied?” The answer depends on where the device that you want to protect resides. This is why I always suggest creating topology maps and other diagrams to visualize your network resources and apply security architecture and mitigation decisions. You can create circular diagrams like the one illustrated below (which is very simplistic, but it gives you the basic idea so that you can then customize it to fit your organizational needs). 
Typically, these types of diagrams include resources that surround a “critical system” (in this case the vulnerable device) that you want to protect. The illustration in the onion diagram above helps you visualize and understand the different layers of protection you can apply within your network to protect affected devices. You can also visualize packet flows and understand how security policies can be applied to each network device to protect critical systems and the infrastructure as a whole. You can identify where you can apply the technologies that enable you to gain and maintain visibility of what is happening in your network, as well as apply security policies and identify “choke-points.” A different example is shown in the diagram below:
You can create these “onion diagrams’ in many different ways, depending on where topologically your vulnerable device resides and how you want to visualize each mitigation and identification technique. The following figure shows a very high-level topology of a typical enterprise network. In this example, a device in the data center (shown in red) is affected by a given vulnerability.
As you can see in this network topology, there are many network devices that can provide different mitigation and identification capabilities. For instance, adjacent switches can provide Layer 2 mitigation capabilities. Cisco ASAs can be configured to only allow transactions from trusted devices or block specific exploit traffic. They can also provide visibility through syslog messages and counter values displayed in the output from show commands that may be included within the AMBs, where applicable. Cisco IOS NetFlow records can provide visibility into network-based exploitation attempts. Some of these techniques and “device roles” are illustrated in the following figure.
Testing Procedures
Testing is critical to the concept of patch management. As part of existing software management and patching procedures, customers may test software upgrades and recommended releases in their environment. Each environment is different and will require different testing procedures. Some customers may select a testing methodology based on the role or nature of the device. For instance, a patch applied to a core router in a service provider network could be tested differently than a patch on an IP phone sitting on someone’s desk or a Cisco Nexus switch sitting at a cloud provider. As an administrator, you’re responsible for testing the new software and making sure that it addresses any problems before your users experience them on their systems. In general, when you patch an application or networking device, you’ll be upgrading a software package. There is no single methodology for this testing. Testing methodologies are as diverse as available applications, services, and devices.
Patches that Require a Reload and Those that Don’t
There are some software upgrades that may require a device reload and other that don’t. For instance, upgrades to Cisco IOS Software, Cisco IOS XE Software and Cisco ASA Software may require a device reload. However, software upgrades to Cisco IOS XR Software may not require a device reload. Cisco IOS XR uses In-Service Software Upgrade (ISSU), which sustains system availability during installation of a software upgrade. ISSUs or hitless software upgrades (HSUs) allow you to upgrade most Cisco router software features without affecting deployed services. You can target particular system components for upgrades based on software packages or composites that group selected features. Cisco pre-configures and tests these packages to help ensure system compatibility.System Maintenance Upgrades (SMUs) deliver software change to the user in the least possible time. Prior to ISSU support, SMU installations resulted in either the restart of one or more processes or a reload of one or more nodes. For more information about Cisco IOS XR SMUs refer to the “Updating Software Images Without a Router Reload” section of Cisco’s IOS XR Release Notes.
Applying Patches and Software Upgrades
Manually upgrading affected devices and applying patches can be error prone and time consuming. In most cases, you need to schedule the upgrades and implementation of network changes. Changes may include published workarounds and mitigation techniques, software image updates, and support for user-initiated ad hoc changes and compliance updates. Tools such as Cisco Prime Infrastructure can help with automating software image deployments when the fix is a device upgrade. Cisco Prime Infrastructure can automatically download the image from cisco.com then deploy the image to one or more devices using a configurable schedule.
Operational Metrics for Patch Management and Device Compliance
Operational security metrics for patch management and device compliance are great tools that could help you better understand your network’s overall security posture. Understanding security operational metrics doesn’t require classes on Nobel Prize-winning theories or complicated math formulas that could make the process too complicated even to execute. You have to understand what you are trying to protect and first establish a high-level process map via your own research. Use common knowledge and a broad survey to validate and identify metrics in each procedure or operational area. For instance, build a set of metrics for things like patch management and device compliance for secure configuration best practices. The following are some of the questions you can ask when thinking about such metrics:
- How long does it take you to be aware of any new vulnerability announcements from Cisco or any other vendor?
- How long does it take to identify affected devices?
- How long does it take to implement workarounds (when available)?
- How long does it take for you to test and implement the fix/patch?
- Do you have devices that are not using a “certified image/version”?
- What percent of devices are in compliance with certified software images?
- What percent of devices are in compliance with standard configuration templates?
The biggest risk in running a “non-certified” software version is the exposure to other software vulnerabilities and non-security issues that can cause network instability. If a new Security Advisory is released with a highly critical vulnerability that may impact hundreds of different products, it will be difficult to identity the impacted devices in a timely fashion. Furthermore, software version control is a best practice while deploying consistent software versions on similar network devices. This improves the chance for validation and testing on the chosen software versions and greatly limits the amount of software defects and interoperability issues found in the network. Limited software versions also reduce the risk of unexpected behavior with user interfaces, command or management output, upgrade behavior and feature behavior. This makes the environment less complex and easier to support. Overall, software version control improves network availability and helps lower reactive support costs.
I always recommend creating standard configurations for each device classification, such as routers, switches, firewalls, and any other security or network device. Each standard configuration should contain the global, media, and protocol configuration commands necessary to maintain network consistency, resiliency, and overall security. You can use several global configuration commands or templates in all devices that are alike and include things such as service commands, IP commands, AAA commands, vty configuration, banners, SNMP configuration, and Network Time Protocol (NTP) configuration. Additionally, make sure to document device and interface “descriptors.” These descriptors include the purpose and location of the interface, other devices or locations connected to the interface, and circuit identifiers. This helps your support and security groups better understand the scope of problems related to an interface and allows faster resolution of problems, such as security incidents.
So Do I Have to Upgrade?
In summary, you do not always have to rush and immediately upgrade a device since network and in-device mitigations may exist. In many cases, upgrading may not even be possible if a patch is not available (for example, when dealing with a zero-day vulnerability). Regardless, the recommendation is that you follow best practices and ultimately upgrade to a fixed version of software, as this is truly the only way to fully remediate a vulnerable device. Remember that mitigations should be considered temporary and they can disappear with network configuration errors and other changes. Please share your thoughts in the comments below on what has worked best for you when responding to security vulnerabilities and security incidents.
.
