Home
Blog
How to avoid Kubernetes misconfigurations

How to avoid Kubernetes misconfigurations

May 9, 2023

Oshrat Nir
Developer Advocate

Kubernetes is an open-source platform that enables the management of containerized applications with ease. The platform is portable and flexible, working across various environments, including Linux, MacOS, and Windows.

A key feature of Kubernetes is the configuration of various resources. These configurations determine how the operations occur, how efficient the complete application’s functioning is, and how secure the application resources are. However, misconfigurations are quite common in the deployment of Kubernetes if recommendations are not followed. Misconfigurations lead to several issues, including vulnerability to attacks and open access to sensitive information. Thus, ensuring that your Kubernetes infrastructure is configured properly becomes crucial for system security.

In this article, we’ll look at the most common misconfigurations in Kubernetes and how they can be corrected. Let’s get right to it.

Impact of misconfigurations on Kubernetes

Here are some of the main concerns that can result from misconfiguration:

Misconfiguration usually increases the attack surface, thus increasing the system’s vulnerability. Often people use default configurations which leads to a security threat because any cyber attacker can access the complete host by using one object or resource as the gateway.

Misconfigurations related to network policies lead to poor traffic management as it doesn’t restrict access to endpoints and ports, which malicious users can use as entry points.

Resources without limitation rules can be “hijacked” by malicious actors for things like crypto-mining or as a backdoor for running malware.

If Kubernetes configurations are not set based on security recommendations, users can easily access sensitive information. They can then modify it or use it to get control of the complete system.

Common types of Kubernetes misconfigurations

These are some of the most common misconfiguration errors:

Privileged containers pose the biggest risk to the host’s security. When a container is given CAP_SYS_ADMIN permissions, it passes the value of allowPrivilegeEscalation to be true. Here, if any user gets access to the container they can access the complete host and take control of it.

cmd.exe or bash must be removed from the container. If this is left in place, it can give users permission to run a shell script within the container. Unfortunately, this type of unauthorized access is a potential backdoor for running malware in the system.

Many times, too often in fact, users get a cluster-admin role. It is an easy solution to an operational problem. Easier than applying the correct role-binding to a user. This is a very serious threat because it gives the user permission to run any command on any resource, which constitutes a major system vulnerability.

DevOps engineers tend to not block network access points through ingress and egress filters. Like the previous misconfiguration, this also creates a security loophole as it doesn’t manage what kind of traffic can access which ports or endpoints.

Configuring immutable container file systems in Kubernetes can help mitigate damage if a cyber attacker gains access to the system. That being said, it still leaves the container vulnerable unless other security measures, such as network segmentation, access controls, and monitoring are also implemented.

Many times container images are used without checking or changing default configurations. This can be a serious problem as default configurations may be permissive and do not follow the principles of zero trust and least privilege.

Limiting unnecessary traffic tends to be overlooked, but can have a serious impact on Kubernetes security. In this scenario, any user can perform a malicious attack and harm the system. This can be prevented by setting network policies so that objects have no unnecessary communications with others.

By default, Kubernetes secrets are stored as base64-encoded strings in etcd, which is a distributed key-value store used by Kubernetes to store its configuration data. Any person can gain access to etcd by accessing the Secret and modifying it. Therefore, encrypting Kubernetes Secrets using tools such as OpenSSL, Vault, SOPS or KMS is essential.

Linux environments are often unhardened, which poses a high chance of a security breach in the cluster. It is therefore necessary to harden the environment using security services such as SELinux, AppArmor, and seccomp. From Kubernetes v1.22, SELinux is enabled by default but should be verified in any case.

Leaving the control plane exposed to an untrusted network is another major mistake engineers tend to make. One should use SSH tunneling between the API server and kubelet or by verifying the kubelet serving certificate.

What We’ve Learned from Scanning 10K+ Kubernetes Clusters

How ARMO Platform solves Kubernetes misconfigurations

Following are the ways misconfigurations can be identified and solved by using ARMO Platform:

Automated vulnerability and configuration scanning

Automated configuration scanning enables the platform to persistently find gaps in the system and alert the engineers who can fix them. After the initial deployment of a cluster, the vulnerability scanner checks container images and flags vulnerabilities. The same occurs when the new container images are added to the system. Thus, security issues and other errors caused due to misconfigurations can be corrected. Subsequent scans can be triggered manually, by an event, or scheduled periodically. All of which increase the chances of finding and fixing misconfigurations as early as possible.

Suggested remediation for risks and vulnerabilities

Based on the vulnerabilities and the misconfigurations flagged by the scanner, ARMO Platform suggests remediations for misconfigurations and fixes for vulnerabilities. These are easily filtered and prioritized through a visual dashboard where the platform guides the users to the necessary fixes.

RBAC visualizer

Role Based Access Control (RBAC) is an important part of Kubernetes configurations. With this, organizations can decide which personnel have what kind of access and privileges in the system. ARMO Platform offers an RBAC visualizer where you can see all the users and their privileges. With the visualizer, such critical information can be easily accessed and queried.

Get your Kubernetes Security Checklist now

Best practices for Kubernetes configurations

Following these recommendations can help in keeping your configurations error-free:

Kubernetes configuration files must be in the Version Control System (VCS). Approval filters must be applied in such a way that if one person makes changes, their peers can double-check those before committing them to a main branch. Furthermore, having config files in the VCS adds to the audit trail of changes and increases the cluster’s security.

Using YAML for writing config files is better than JSON because the former is more humanly readable. Thus, different team members can easily understand the configuration and fix it if necessary.

All connected objects must be grouped into a single file because it is easier to manage, troubleshoot, and perform operations on a single file as opposed to multiple files.

Do not use default values, as misconfigurations are bound to lead to loopholes and gaps in the Kubernetes infrastructure.

Network policies determine which objects can communicate with each other and which objects can be accessed by the incoming traffic. Although, by default, all objects can communicate with each other, engineers must change this and put rules of access in place. Thus if a malicious attack occurs, the attacker will not be able to move across multiple objects, due to the small attack surface.

To restrict internal traffic, one must also set up firewalls. Thus, only IPs in an allow-list of IPs can send requests to the API, and not everyone is able to access the cluster.

Summary

Overall, misconfigurations are one of the most common reasons behind Kubernetes clusters’ security breaches. Although numerous misconfigurations can occur, using solutions such as ARMO Platform and Kubescape and following best practices above can help you avoid misconfigurations. This results in a hardened Kubernetes cluster with a smaller attack surface.