CVE-2022-47633: Kyverno’s container image signature verification can be bypassed by a malicious registry or proxy

Executive summary

Security researchers at ARMO have found a high-severity vulnerability in the Kyverno admission controller container image signature verification mechanism. The vulnerability enables an attacker who is either running a malicious container image registry or is able to act as a proxy between the registry and Kyverno, to inject unsigned images into the protected cluster, bypassing the image verification policy. 

The vulnerability was introduced in version 1.8.3 and was fixed in version 1.8.5. We recommend all Kyverno users update as soon as possible.

Background

Supply chain security is a thoroughly discussed topic in the Cloud Native community. It comes as no surprise that projects like Kyverno have started to incorporate supply-chain security measures into their offerings.

In order to secure supply chains, validation of software components should happen at every step from development to production. Container images are an obvious potential attack vector, so it is critical that they are validated to ensure that only verified code is running in a Kubernetes cluster. If an attacker is able to control the contents of an image that is pulled into the cluster, the attacker can effectively take control of a victim’s pod and use all of its assets and credentials, including the service account token to access the API server.

In order to protect a Kubernetes cluster against malicious container images, security-focused admission controllers offer policies that can enforce container image signatures. Their promise is that with an admission controller, users can create a policy to only have validated images running inside the cluster.

Image signature validation in admission controller

Here is the way signature validation works in the Kyverno admission controller.

1. A new workload, defined via an image with a tag, is requested from the Kubernetes API server.
2. The API server asks the admission controller to validate the new workload. The admission controller controls whether a workload can be started on (“admitted to”) the cluster.
3. The admission controller, configured to validate the image signature, requests the image manifest and the signature from the container registry.
4. The container registry supplies both the signature and the image manifest itself. 
5. Depending on whether the image manifest and the signature are correct, the admission controller allows the new workload into the cluster. To prevent spoofing, Kyverno changes the image to be pulled by its cryptographic hash, and not its tag, which is mutable.
6. The API server asks the Kubelet to start the new workload.
7. The Kubelet asks the container runtime to start a new container based on the image and hash from step 5.
8. The container runtime downloads the image manifest and its layers from the container registry.
9. The container runtime starts the new container based on the image.

The attack

An attacker’s goal may be to inject an unsigned image into a cluster which should only be running signed images.

In this attack, we assume the attacker controls a malicious container registry or has set up a proxy between the registry and the target. The attacker attempts to trick the user into running a Pod with an image from this registry. Meanwhile, the cluster administrator has a policy in place to protect the cluster from malicious images by enforcing container signatures. From the cluster administrator’s perspective, no unsigned images can be admitted into the cluster.

The attack steps are as follows:

1. The user is convinced to run the signed image from the malicious container registry, or to use a malicious proxy to download images. 
2. The API server asks the admission controller for approval.
3. The admission controller asks for the image manifest based on the image tag. Based on the manifest, it gets the image hash and asks for the signature for the signed image based on image hash.
4. The malicious proxy returns the signed image signature to the admission controller.
5. The admission controller verifies the signature of the signed image.
6. Due to a bug in Kyverno, the admission controller requests the manifest of the signed image for a second time to get the digest for mutation.
7. The malicious registry returns the manifest of a different image – this one unsigned and malicious.
8. The admission controller changes (mutates) the image in the Pod spec from image tag to image hash and gives approval to the API server.
9. The Pod starts, running the unsigned image.

The problem is that the image manifest is downloaded twice. It is pulled once for signature validation and a second time for mutating the image name in the Pod spec. This is a classic example of a TOCTOU problem that allows the attacker to pull a bait-and-switch. Since the image manifest which will eventually be used is not the same as the one which was verified, this enables the attacker to trick the client.

Summary

Kyverno implements a mechanism to verify container images against supply chain attacks. It automatically substitutes the image tag for the digest on first use. However, a bug introduced in Kyverno 1.8.3 can cause multiple tag-based queries to the image registry, which can be exploited by attackers. This problem has been fixed promptly by the Kyverno team, in version 1.8.5, and users should update to the latest version. The fix ensures that the same image hash is used to change the workload specification as was used to verify the signature. 

Additional recommendations: 

• Implement a policy that only allows trusted registries, which would prevent the malicious registry from being used. 
• Use Kubescape to scan container image registries in order to make sure you’re not pulling malicious images.
• Kubescape has developed a dedicated control – C-0091– in the ARMOBest framework verifying if this CVE exists in your cluster. Please install or update to the latest Kubescape version from GitHub, or via the following command:

curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash kubescape scan control C-0091