Kubernetes RBAC: Deep Dive into Security and Best Practices

What is RBAC in Kubernetes?

Kubernetes has revolutionized container orchestration, becoming the go-to platform for managing containerized workloads at scale. However, with its growing popularity, the complexity of managing role-based access control (RBAC) on Day 2 and especially in a multi-cluster environment has become a daunting task. DevOps, SRE, and Platform teams are responsible for multiple clusters and different teams. This article explores the challenges of RBAC implementation, best practices for managing RBAC in Kubernetes, and the innovative solution offered by ARMO Platform to simplify the use of RBAC and enhance Kubernetes security.

Understanding RBAC in Kubernetes

RBAC is a fundamental security mechanism in Kubernetes that governs access to resources within a cluster. RBAC policies define which users or groups can access specific Kubernetes resources and the actions they are allowed to perform on those resources. The RBAC process involves three steps: authentication, authorization, and admission control.

Authentication

Authentication is the process of verifying the identity of a user or entity. There are three types of entities in Kubernetes: a user (which is usually a human), a group (which is usually a set of humans), and a service account (which is used by pods inside the cluster). Users need to be created using X.509 certificates signed by the cluster CA, which can be done through CSRs and certificate issuing. How this is done is outside the scope of this document, but there are many online tutorials explaining how to do this.

Pods that are running inside the cluster don’t need a certificate. Instead, a service account can be assigned to them. When you create a pod without specifying any account for it, the “default” service account is automatically assigned to the pod. User accounts are also available for processes that run inside pods. You don’t create an X.509 certificate for them; instead, you link pod and user accounts, which is typically done in manifest files. For example:

apiVersion: v1

kind: ServiceAccount

metadata:

name: my-service-account

In the RBAC process, authentication ensures that the user or entity requesting access to the Kubernetes cluster is who it claims to be. Once the entity is authenticated, the next step is authorization. 

Authorization

Authorization determines what actions a user or entity is allowed to perform within the Kubernetes cluster. Once a user or entity is authenticated, the RBAC system checks their permissions and RBAC roles to determine if they have the necessary privileges to perform the requested actions and access rights. This step ensures that only authorized users can perform specific operations within the cluster.

Admission Control

Admission control is a mechanism in Kubernetes that intercepts and processes API requests before they are persisted in the cluster. It enforces policies and rules to validate and modify the requested resources. In the RBAC process, admission control plays a role in ensuring that the requested actions comply with the defined authorization policies. It can reject or modify requests that do not meet the specified criteria. 

It is important to note that RBAC is not necessary for Kubernetes resources specified in manifest files. For example: let’s say you want to create a pod and a secret and give the pod access to the secret. The pod specification in the manifest file specifies that the secret must be “mounted” one way or another (through environment variables or files actually mounted inside the pod). In this example, the pod can access the secret no matter the RBAC permissions assigned to the pod’s service account. This is because the mounting of such secrets does not depend on RBAC or metadata.

Anatomy of RBAC rules

An RBAC rule is made up of three elements: the API group, a verb (i.e., an action), and a target (either a resource name(s) or an API URL). RBAC rules are specified in roles and cluster roles (the difference between the two is that roles are scoped to a given namespace, while cluster roles apply to the entire cluster). This is what an RBAC rule looks like, as part of a YAML file specifying a role or a cluster role:

- apiGroups: [“”]

verbs: [get, list]

resources: [secret]

resourceNames: [mysupersecret]

The API group identifies which API group to target. This is necessary because different API groups can have the same verbs. Additionally, Kubernetes is highly extensible and allows for the addition of new APIs that can have verbs and resource names that clash with other APIs. In the manifest file, this is a list, although usually only one API group is specified here. The API group “” (empty string) represents the core Kubernetes API.

The verb indicates the action to take. For example: get, list, create, delete, update, etc. Again, this item in the manifest file is a list, which allows you to specify more than one action and thus avoid repeating very similar rules over and over again.

The third part is the verb’s target. It can be a resource, which is explicitly specified. Examples of resource types are: pod, networkpolicy, service, etc. Additionally, further restrictions can be applied by using resource names.

The second option for the verb’s target is to specify the URL path. This is the request’s URL path and can contain an ending wildcard, which can be used to give access to certain parts of the API. An RBAC rule must specify a target that is either a resource or a URL, but not both. Also, please note that a URL path can be specified as a target only for cluster roles.

Examples of RBAC rules

RBAC with a resource target

Example YAML manifest file:

apiVersion: v1

kind: Role

metadata:

        name: test-role

        rules: - apiGroups: [“”]

        verbs: [get, list]

        resources: [secret]

        resourceNames: [mysupersecret]

The role in this file has only one RBAC rule, which targets the core API (the empty string in “apiGroups”). The rule allows for the actions “get” and “list” on a secret named “mysupersecret.”

RBAC manifest file with a URL path

Example of YAML manifest file:

apiVersion: v1

kind: ClusterRole

metadata:

        name: test-cluster-role

        rules: - apiGroups: [“rbac.authorization.k8s.io”]

  verbs: [get, list]

  nonResourceURLs: [/apis/rbac.authorization.k8s.io/v1alpha1]

In this case, the rule gives “get” and “list” access to any request made on the “/apis/rbac.authorization.k8s.io/v1alpha1” path. Please note that the role here is a ClusterRole, as only cluster roles can use path-based permissions.

Accessing cloud services

If you want your pods to access cloud services, RBAC is not enough because it only manages Kubernetes resources. In AWS, for example, you will need to link a Kubernetes service account with an IAM role. Achieving this is a somewhat complex process, which is well-documented by AWS. In short, you will need to add annotations to your service account manifest that point to the IAM role you want the service account to assume with respect to accessing cloud services and preventing attackers from bypassing RBAC mechanisms through node access. For example:

apiVersion: v1

kind: ServiceAccount

metadata:

    name: my-service-account

   annotations:

         - eks.amazonaws.com/role-arn: <ROLE_ARN>

However, you will need first to create an OIDC provider (one per EKS cluster). Other cloud vendors require similar intraicacies in order to access their resources.

RBAC best practices

Effective RBAC management requires adherence to best practices to ensure the security and integrity of the Kubernetes environment. Here are some RBAC best practices:

Principle of Least Privilege

Apply the principle of least privilege to service accounts to grant the minimum necessary permissions required for each user or group to perform their tasks. While applying the same principle to humans is ideal, it can be more challenging to achieve in practice.

RBAC Strategy and Planning

Create a well-defined RBAC strategy, starting with an assessment of the current state and defining clear goals. Develop a plan to transition from the current state to the desired RBAC policies, considering the specific needs of your organization.

Infrastructure-as-Code and Templating

Use infrastructure-as-code (IAC) or templating tools like Helm charts to manage RBAC policies efficiently. Templating enables you to use variables and keep track of changes, aiding in troubleshooting and auditing processes.

Comprehensive Testing

Regularly test RBAC permissions to ensure that entities can perform their necessary actions without being allowed to perform unauthorized actions. Thorough testing helps identify and rectify vulnerabilities in the system.

Periodic Review and Cleanup

Periodically review existing RBAC and service account objects to ensure they are up-to-date and delete those that are no longer needed. This minimizes the attack surface and streamlines RBAC management.

RBAC in Kubernetes: Day 2 operations

RBAC policies aren’t static. As mentioned above periodic review and cleanup is a best practice for RBAC. But how?

The most straightforward way to manage RBAC policies is by using the kubectl command-line tool to view, create, update, and delete them.

Here are a few examples of kubectl commands for viewing RBAC policies and related resources:

• kubectl get roles lists all the roles defined in the cluster, including their associated rules and scope.
• kubectl get rolebindings lists all the role bindings in the cluster, which define the mapping between subjects (users or groups) and roles.
• kubectl describe role/<role-name> provides detailed information about a specific role.
• kubectl auth can-i <verb> <resource> checks if the current user has the specified permissions (verb) for a specific resource.

In addition to these commands, there are others to add, remove, and delete RBAC policies. 

Although kubectl provides an easy-to-use operational experience, there are some drawbacks, which we discuss below.

Cluster scope

Kubectl commands are executed for a single cluster only. This means that when you’re managing RBAC policies across multiple clusters, tracking which policies have been updated and which ones need to be updated can be challenging. 

Lack of holistic view

The commands mentioned above show the RBAC-related Kubernetes resources in a CLI format, such as a cluster role to watch deployments:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-watcher
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]

The output shows the definition of the cluster role with its rules. What it lacks is contextual information, like who’s using a role within a cluster. To find that out, you also need to check cluster role bindings, as well as role bindings for cluster-wide and namespace-scoped access. 

So using kubectl commands is not enough to create a holistic view since they provide very granular information. 

Challenges of managing RBAC in a multi-cluster environment:

As organizations adopt Kubernetes and deploy multiple clusters to manage their containerized workloads, RBAC management becomes more complex. Several challenges arise in managing RBAC across multiple clusters:

Role Explosion

In a dynamic environment with numerous users and roles, it becomes challenging to manage granular access control. Each user or group may require unique permissions, leading to a growing number of roles, making it harder for administrators to keep track of privileges.

Complexity and Scalability

As the number of roles and permissions increases, the complexity of managing relationships between different roles and permissions within clusters becomes challenging. Without native tools for synchronization, administrators may struggle to comprehend the overall access control system.

Lack of Holistic View

The command-line tools like kubectl provide granular information about individual roles and bindings, but they lack a comprehensive view of how different roles are used across the clusters.

Imperative Execution

Imperative RBAC management through kubectl commands lacks continuous synchronization and checks, making it prone to human errors and time-consuming updates.

Conclusion

RBAC management is crucial to securing a Kubernetes cluster and ensuring compliance with regulations and industry standards. However, without the proper tools in place, managing RBAC can be a complex and error-prone manual process. This is why RBAC visualization is essential. Visualizing RBAC helps administrators to keep track of all privileges in a systematic and organized manner. A visual representation of the relationships between roles and bindings allows administrators to understand the policies more easily, and thus make informed decisions about policy changes. RBAC investigation is critical to securely managing Kubernetes in production and ensuring security and compliance.

ARMO platform includes an RBAC visualizer allowing administrators to see which privileges are assigned to any given user. With ARMO Platform, administrators can take control of RBAC management and reduce the attack surface by conforming to the principle of least privilege. Try it today, it’s free!