Under the hood of CVE patching

Addressing Common Vulnerabilities and Exposures, known as CVE patching, is a practice of applying updates to software (patching) to address security vulnerabilities. CVE patching is your shield against the threat of malicious actors exploiting such weaknesses and is of crucial importance for every organization’s cybersecurity.

This post will cover the basics of CVE patching: the roles and stakeholders, the step-by-step process, and common mistakes to avoid. 

Industry standards: CVSS, EPSS, and KEV

When dealing with vulnerabilities and patches, you’ll encounter certain industry standards. Let’s quickly explore the most prevalent ones.

The Common Vulnerability Scoring System (CVSS) offers a standardized method for measuring just how serious a vulnerability is. Using a score of 0 to 10, it lets you rank vulnerabilities to prioritize your security efforts. 

On the other hand, the Exploit Prediction Scoring System (EPSS) estimates the risk that a vulnerability will be exploited. It produces a score of between 0% and 100%, with a higher score indicating a higher probability of exploitation.

Finally, the Known Exploited Vulnerabilities (KEV) is a catalog of known vulnerabilities compiled by CISA (Cybersecurity and Infrastructure Security Agency). It currently has over 1,000 vulnerabilities and is continuously updated. 

The CVE patching ecosystem: key stakeholders and their roles

CVE patching is a collaborative effort by various stakeholders, both within and outside an organization. A seamless and secure patch management process is only possible if the roles and responsibilities of all players are properly understood.

Internal stakeholders

Internal stakeholders are those involved in the CVE patching process within an organization, mostly, its employees.

Security Operations Team and Security Operations Manager

The security operations team, often led by a security operations manager, is at the forefront of CVE patching. They are responsible for identifying vulnerabilities, prioritizing them based on severity (using frameworks like CVSS), and formulating a patching strategy. Their proactive approach ensures that the organization stays one step ahead of potential threats and that the most important patches are applied first.

This team is also charged with providing mitigation to vulnerability risk when a patch is not yet available for the vulnerability. For instance, risk mitigation could be achieved with firewalling, intense monitoring, service disabling, or applying security context. When that mitigation has been chosen, they also are responsible for signing off on the “risk acceptance.”

System Administrators or DevOps

These professionals are in charge of the actual patch deployment and ongoing monitoring. They work collaboratively with the security operations team to ensure a smooth patching process that minimizes system disruptions. System administrators or DevOps teams play a crucial role in maintaining system stability.

QA and Compliance Teams

Quality assurance (QA) and compliance teams are responsible for testing patches to make sure they fix problems and do not introduce new issues. They also play a significant role in verifying that the organization remains compliant with industry regulations throughout the patching process.

External stakeholders

External stakeholders are professionals involved in the patching process who work outside the organization.

Vendors and Developers

Software vendors and developers are in charge of developing and releasing patches for their given software, which may be used inside your organization. When vulnerabilities are discovered, they develop and release a patch fixing the issue. They also make sure that you are informed of the vulnerability and its patch.

Vulnerability Database Providers

These organizations maintain databases of known vulnerabilities, working closely with security researchers to document and spread information about them. Access to these databases is crucial for identifying and understanding vulnerabilities in the first place.

The CVE patching process: safeguarding your digital ecosystem

CVE patching is a precise process designed to strengthen your digital environment against potential threats. Organizations must understand the steps of the CVE patching process so that it can be implemented effectively and minimize the risk of vulnerabilities. When the mean time to remediation (MTTR) of an internet-facing vulnerability is 58 days, an organization must be ready to react quickly by being aware of the process.

1. Vulnerability assessment

The journey begins with a comprehensive vulnerability assessment, typically made by the security operations team. This involves identifying security weaknesses within your software and your systems. 

Keep in mind that there are typically many vulnerabilities and possible issues that will emerge from a vulnerability assessment. This can create information overload, or noise, making it challenging to filter out the most important ones. Vulnerability scanners, manual assessments, and vulnerability databases can help identify the vulnerabilities most in need of attention due to their potential impact. 

This critical first step forms the foundation of your patching strategy.

2. Prioritization of vulnerabilities

Once you have a list of the existing vulnerabilities, you can then prioritize them. Vulnerabilities will have different levels of severity, and addressing them all simultaneously may not be practical. 

Prioritization is key. You will need to utilize frameworks such as CVSS, EPSS, and KEV (often in conjunction) to rank each vulnerability according to its potential impact and exploitability. Factors including your industry, company size, vulnerability reachability, and the criticality of the system, as well as whether it’s internal or external, should also be taken into account. 

There are tools that can help with vulnerability prioritization and reducing information noise. ARMO Platform provides an additional filter, giving you a shortlist of the most imminent threats.

3. Patch acquisition

With a prioritized list in hand, it’s time to acquire the necessary patches from software vendors or trusted sources. Ensure that the patches are official and come from reliable channels to avoid introducing malicious code into your systems inadvertently.

In some cases, a patch may still be under development, especially for newly discovered vulnerabilities. Make sure to keep an eye on the progress and release dates from the vendor or developer. In the meantime, consider alternative measures to mitigate the risk temporarily, such as increased monitoring or service shutdown/replacement.

In some cases, patch acquisition won’t be necessary, as updating to the new image that has fixed the vulnerability is the way to go. Which itself is not a straightforward process. It requires going through the steps detailed below, before being applied across all environments.

4. Testing in non-production environment and patch validation

Validation is key to making sure no new vulnerabilities are inserted and the old ones are patched. You can use ARMO Platform’s image scanning capabilities to be sure. Before applying patches to your production environment, it’s important to test them thoroughly in a controlled, non-production environment. This is where the QA team should be the most involved. Testing helps find any potential conflicts, performance issues, or unexpected consequences that might arise when the patches are applied to live systems. 

5. System backup

As a precautionary measure, the IT team should perform a full backup of your system before patch deployment. This way, if there are any issues during or post-patching, your system can be restored to its previous state. 

6. Patch deployment

Once the patches have been tested and you have a backup in place, it’s time for the patch deployment. Carefully follow a predefined schedule, considering factors such as maintaining system availability and minimizing service disruptions. Consider deployment strategies like ramped, blue/green, and canary to ease the transition. Be prepared for a possible brief period of downtime during this phase.

7. Monitoring and validation

Continuous monitoring is vital after a patch is deployed. Watch for any anomalies, system errors, and performance issues. The DevOps and IT teams can work together to make sure all systems are working correctly, the patches were applied correctly, and the vulnerabilities have been successfully mitigated.

8. Communication and documentation

Clear communication is essential throughout the CVE patching process. Keep stakeholders informed about the progress and outcomes, including internal technical teams (DevOps, IT, security ops, etc.) as well as non-technical teams if relevant. 

Maintain detailed documentation of the entire process, including the vulnerabilities addressed, patch deployment details, testing results, monitoring logs, and risk acceptance and mitigation that has taken place outside of the K8s infrastructure. This documentation is critical for compliance, audits, and future reference.

What companies get wrong when patching CVEs

While addressing CVE patching is essential, companies can hit some unnecessary hurdles. Read on to make sure your organization avoids making these mistakes.

Delayed patching

One of the most widespread mistakes in CVE patching is delayed patch application. Many organizations or even individuals tend to procrastinate when it comes to applying security updates. While your systems are unpatched, they are vulnerable to known threats, which cybercriminals actively target.

Procrastination can be caused by a phenomenon known as “CVE shock,” which is the overwhelming feeling that emerges when facing a large number of vulnerabilities.

To avoid delayed patching, establish a structured patch management process with regular vulnerability assessments and a predefined schedule for patch deployment. Additionally, stay informed about emerging threats and software updates to make sure patches are applied in a timely manner.

Lack of testing

Insufficient testing before patch deployment is another significant barrier to successful patching. Rushing to implement patches without proper testing can result in major issues such as service disruptions to customers or loss of data, with a patch ultimately creating more problems than it solves.

Effective testing involves creating a controlled environment that mirrors your production systems. You want to test the patch with conditions as similar as possible to production so that any issues are reflected. Only after thorough testing and validation should patches be applied to the production environment.

Ineffective prioritization

Not all vulnerabilities are created equal, and determining which ones to address first is very important. 

Getting prioritization wrong can lead to critical vulnerabilities being left unaddressed while less urgent ones are patched. This creates gaps in your security strategy, meaning your systems will be left exposed to hackers.

To address this challenge, consider using CVSS and EPSS, as well as other risk assessment frameworks, to assign a severity rating to each vulnerability. Develop a risk-based approach to prioritize patches, focusing on the most critical and exploitable vulnerabilities first. The end goal is to minimize the risk of allowing hackers into your systems. ARMO Platform offers the ability to filter according to score, but also according to reachability, to flag those vulnerabilities that are easily accessible to malicious actors.

Improperly managed downtime and disruptions

Applying patches can result in downtime or service disruptions that affect your users. While this is a necessary part of maintaining security, it must be managed carefully to minimize its impact.

To mitigate the effects of downtime and disruptions, consider performing patching during off-peak hours or scheduling it in a way that minimizes any interruption of essential services. Also, a rollback plan will help you restore systems to their previous state if issues arise during patching.

Summary

CVE patching is an end-to-end process to ensure that your system and software are up to date with the most recent and secure version of software.

It is a collaborative effort between internal and external stakeholders. Industry standards such as CVSS, EPSS, and KEV are key to estimating a vulnerability threat and keeping up to date with the latest vulnerabilities. However, looking at the context of your environment and runtime needs allows for even more precise prioritization and increasing signal-to-noise ratio. Learn how ARMO Platform can strengthen your Kubernetes security posture with contextual vulnerability assessment and reduce CVE-related prioritization effort by more than 50%. Thus improving the overall efficiency of your vulnerability management.